One of the most difficult components of understanding and working with AWS is learning how to deal with the extensive and robust networking side of things.

From the bare basics of working with route tables and CIDR blocks, all the way up to connecting VPCs across regions - You will have to learn a lot about getting things connected within the cloud.

One of the most troubling parts of any architect’s or engineer's job is trying to create a well-architected method of connecting multiple networks together within AWS, in the most secure and efficient way possible. 

Now, by default, the VPC is supposed to be an isolated selection of the cloud made private for our sole use. This means of course that when you add an EC2 instance within that framework, it shouldn't, and can't, communicate directly with another network without some kind of additional force acting upon the system.

But let's say you did want to add that force, that you wanted networks to be able to speak with each other, what could you do?

Well, The good news is that at a high level, connecting two networks within AWS is actually fairly simple. You just need to hook them up using a VPC peering connection, which will allow you to transfer data between those entities.

VPC peering is a secure and relatively painless method of extending your route propagation into another network. There are some catches you have to be aware of but otherwise, it works as intended.

You might also think about how you can connect your on-site networks with your networks within the cloud. This is also easily achievable and can be done using a VPN connection, which will create a secure tunnel between your on-premises environment and AWS.

Both of those two methods are extremely common and are used by a large majority of AWS users. Where the problem starts to come in, is when we have tens or hundreds of these VPCs or onsite connections that we need to make and maintain.

There have been various models over the years that can help deal with this problem, like the hub and spoke model if you have a bunch of networks that all needed access to one specific VPC. 

This would be common where that single VPC is a security watchdog and needs to inspect all traffic before it heads out onto the internet or checking incoming packets for malicious intent, and then dispersing them out to the appropriate VPC.

This kind of setup is quickly obtainable using VPC peering as we discussed before - but starts to become tedious and even troublesome to maintain as more and more networks need to join the fold.

Imagine a use case where there are 5 VPCs and each one needs to know about the other. If you were to set up a 1 to 1 VPC peering connection between each of them that would be 10 separate VPC Peering connections you would need to set up and maintain.

As the number of connected networks increase, so does the complexity involved with maintaining those connections. 

Today I want to discuss the best way that amazon has created so far to deal with these many connection-related scenarios - and that would be, by using AWS Transit Gateway. 

Transit Gateway can best be described as a ‘network transit hub’ or even as a ‘Regional Router’. It is a service that allows you to interconnect hundreds of VPC and on-premises networks together without all the extra burden that comes with the standard-issue VPC peering or VPN connection maintenance and setup. You can also add direct connect into the system for more reliability and higher speeds when dealing with your on-premises networks.

With this kind of power, Transit Gateway allows you to create some incredibly complex systems and accomplishes some interesting network engineering feats, without you really having to get too deep into wiring the system by hand.

Let’s get started with the basics - Transit Gateway is a managed service. This means that you don't have to deal with any of the underlying features like updating software or managing infrastructure. 

Being a managed service also means that Transit Gateway deals with any scaling concerns that might occur for you. This means when traffic starts to increase, Transit Gateway will add more and more backend power as required - and the same happens in reverse when that traffic dwindles off. This interaction is very similar to how internet gateway operates, scaling elastically based on the demand put on it.

A new Transit Gateway can be created inside the VPC console and is fairly simple to get provisioned. Bear in mind that Transit Gateway is also a regional service, so make sure you are in the appropriate location where you wish to connect your networks.

Each Transit Gateway has an associated route table that helps to direct traffic through it. In fact, your Transit Gateway can contain multiple route tables as well, helping you to mentally and logically separate elements as you wish.

It is important to note that Transit Gateway operates on Layer 3, which means that packets are sent to specific next-hop attachments - based on their destination IP addresses.

We can get traffic into our Transit Gateway by attaching various entities to it. When you attach something - we are creating an attachment point in those objects

A Transit Gateway attachment is both a source and a destination for packets.

You can attach the following with your Transit Gateway.

One or more VPCs.

One or more VPN connections.

One or more AWS Direct Connect gateways.

One or more Transit Gateway Connect attachments.

One or more Transit Gateway peering connections.

At this point in time, these are the only ways you can connect with and pass traffic through the Transit Gateway. 

Let’s go over these one at a time to learn more about the service.

This use case is probably the most common one for many who will use the service. 

Let’s say that you have three different VPC that you wish to connect using Transit Gateway. And in this scenario we want each VPC to be able to talk with each other and pass traffic amongst themselves.

For our example, all of these VPCs will also be in the same region. Transit Gateway can work with VPC in different regions, but there is an extra step to make that happen, and we will cover that a little later when we discuss Transit Gateway Peering.

Transit Gateway requires you to create attachment points within the VPCs you want to connect with. When attaching a VPC to your Transit Gateway, you will be asked to enable one or more availability zones that can be used by the gateway to send traffic into that VPC. 

When you do this, you are allowing the Transit Gateway to place an elastic network interface (an ENI) into a subnet you specify within that availability zone -  this ENI will use a single IP address from that subnet, so make sure you have room if your subnets are small.

This elastic network interface is where all your traffic for that available zone will be sent. This traffic however is not just limited to that specific subnet but will siphon all the traffic for all the subnets within that availability zone.

So it’s important to state then: without this specific network interface within an available zone/subnet, no traffic can flow from any resources within that AZ to the Transit Gateway itself.

It is of course highly recommended that you enable Multiple availability zones. This allows Transit Gateway to place additional network interfaces within those AZs - creating a highly available network.

Amazon mentions there are a couple gotchas you need to be aware of when connecting VPC with Transit Gateway:

“A Transit Gateway does not support DNS resolution for custom DNS names of attached VPCs set up using private hosted zones in Amazon Route 53. “ 

If you do need something along those lines - go check out this AWS post.

And another gotchya:

“You cannot create an attachment for a VPC subnet that resides in an AWS Local Zone.”

It is also important to mention that even though your VPC can be connected through AWS Transit Gateway, that does not mean you can access or reference security groups from one VPC that are in another VPC

Now to actually begin routing your traffic into the Transit Gateway - we need to adjust our route tables within the VPC.

Here is an example route table for one of our VPCs.

Currently, all traffic within our VPC will stay within the VPC. There is no route out, and all local traffic stays here.

But after we have added our Transit Gateway Attachment point, we can start to send traffic through it to the Transit Gateway itself. To do so, just add the following route into the VPCs route table

This new route is specifying that any other traffic that is not going to a local IP address should be pushed over to the Transit Gateway Attachment - I have represented this as TGW-XXXXXX to show that it will have a unique identifier.

Each VPC that you want to send traffic to the Transit Gateway, will need to have a route table similar to this one - so let's add those other two into the system

Now that we have all the traffic being sent from our external VPCs, which we want to talk with each other, we need to now update the Transit Gateway and specify the type of connectivity we require.

Transit Gateway has its own route tables that help to direct traffic through the system. 

For our current example, we will be using a single table but later on, I will show you an example with multiple route tables.

Starting off your Transit Gateways route table will be empty - all of your VPCs are sending their traffic in and have nowhere to go. Let’s fix that.

And it is as simple as that - just tell the route table to send all the packets that come in looking for a specific destination VPC and send them out to that attachment point.

With just this simple route table - we now have interconnectivity between all of our VPCs. The best part about this whole setup is that if we need to add another VPC into the sharing fold - it is as simple as attaching it to Transit Gateway and adding a route to that attachment point within that VPCs route table.  Just make sure to adjust the Transit Gateways route table to allow the original VPCs to send traffic to the newcomer.

If you wanted this same interaction with just plain VPC peering you would have to establish a new VPC peering connection three times. One for each of the other VPCs. and if you were to add another VPC into the fold, that would be four new connections you have to create and maintain. You can see how quickly this can get out of control 

And I think you can also see how scalable Transit Gateway is in comparison to the old system.

Now you might be asking yourself: am I allowed to connect VPCs that have the same CIDR range? The answer is no. It’s just like normal VPC peering in that way.

In our previous example, we were able to connect multiple VPCs into the Transit Gateway and each had the ability to send packets to the other. 

Now I want to take a look at how you can add an external source ( like a VPN  connection) into the mix. 

You can create a link between an onsite customer gateway and AWS Transit Gateway in the console or the command line. This is done by creating a new VPN attachment within your Transit Gateway and selecting the appropriate on-premises customer gateway or by creating a new customer gateway on the fly. If you create a customer gateway like this - you will need to have a static public IP address to use, and your BGP ASN

Finally, for your routing options, you will need to choose between static or dynamic routing.

AWS says the following about this choice:

“The type of routing that you select can depend on the make and model of your customer gateway device. If your customer gateway device supports Border Gateway Protocol (BGP), specify dynamic routing when you configure your Site-to-Site VPN connection. If your customer gateway device does not support BGP, specify static routing.”

Seems fairly straightforward.

Alrighty, now we have a connection between our on-premises site and our Transit Gateway, we just need to get a few routes set up to complete the addition.

Let's go back to our 3 VPCs that are all connected together -  we now want them to be able to talk with the added VPN connection.

By simply adding this route here to the bottom of the already existing route table.

We can send all traffic that is not directed to a specific VPC out through the VPN connection. 

This kind of setup is most commonly used for security purposes - where some onsite process could be checking all of the packets passing through the network. This traffic could then be sent on its way to either the internet or another internal network on site.

We can of course do this exact same treatment with a direct connection, however, it is a little more complicated.

Obviously, you will need to have set up a direct connection to AWS through one of the Direct connect partners around the world.

Next, you need to create a new Direct Connect Gateway within direct connect.

Next, we need to create a Transit VIF ( a virtual interface ) - which is a VLAN that helps to transport traffic from the direct connect gateway to a Transit Gateway.

Finally, we can associate our Transit VIF with our Transit Gateway within the direct connect console. Bear in mind you need to know the network prefixes that you want to advertise to the Direct Connect Gateway.

And there you have it. You can use both a VPN connection and a Direct Connection for redundancy. 

Now for these examples, we have been creating a network that is completely flat. All parts of it have the power to connect with and sent traffic to every other part. What should we do if we only want partial connectivity?

Maybe I don't want my VPCs to be able to send traffic to each other. Well, we can accomplish that fairly easily by just changing a few rules in the route table.

Well to be specific, we need to create another route table to help separate our traffic flows.

Let's create one route table that sends all traffic it receives straight out to the VPN.

And we can associate all of our VPCs traffic with it. This means that there is no route from one VPC to another VPC. Traffic can only flow out of the system.

Now let’s create another route table for the traffic that is flowing inwards from the VPN and send it to the appropriate attachment points.

With this setup, we have created traffic flow from the VPN to the VPCs and back. However, there is no traffic allowed between the VPCs.

Pretty cool!

One of the nicest things about Transit Gateway is the ability to have these multiple route tables. It allows you to create some very impressive network topologies that previously might have been too difficult or cumbersome to deal with.

A question you might have at this point is - how does Transit Gateway determine which path to select when sending a packet? For example what if we have both a direct connection and a VPN setup back to the on-premise network which one does it choose?

Well, there is an order of operation, and it is pretty simple.

Number one, It’s going to check for the Most specific route / longest prefix match.

Then it will check the Static route entries, including site-to-site VPN routes ( static is preferred over dynamic ) - which is a slight difference in behavior over the virtual private gateway.

Third, it will check VPC propagated routes.

Fourth, BGP propagated routes from AWS Direct Connect Gateway.

And finally, BGP propagated routes from AWS site-to-site VPN.

All of the previous examples of using Transit Gateway have been shown while all the VPCs have been within the same region. 

Now, we live in a very latency-caring world these days - so you will probably have your content distributed all over the world and in multiple regions if you have built a well-architected and robust network.

By default, Transit Gateway does not allow you to send traffic to VPCs in different regions.

The good news is that it does allow you to send traffic to other Transit Gateways that are in other regions - effectively peering connections between regions.

This is accomplished by creating a Transit Gateway peering attachment. The Transit Gateway peering attachment allows you to peer two Transit Gateways and this connection supports both IPv4 and IPv6 traffic between.

This connection can be used either with a single account or even with a different AWS account - which is particularly useful for dealing with large organizations where teams will be separated with different accounts on purpose.

To begin peering your gateways create a peering attachment on your Transit Gateway and specify another gateway within a different AWS region.

From there, the owner of the requested gateway (which can also be referred to as the acceptor Transit Gateway) has to accept the request.

To actually begin routing traffic over to the other gateway, you need to add a static route in your route table that points to the Transit Gateway peering attachment.

AWS also says the following - ”We recommend using unique ASNs for the peered Transit Gateways to take advantage of future route propagation capabilities.”

Another thing to note is this Transit Gateway peering connection does not support resolving the public IPv4 DNS hostnames to private IPv4 addresses.

If you already have an on-premises SD-WAN setup and are looking to add a connection into the AWS cloud - Transit Gateway has a special connection option already set up for just this scenario.

AWS Transit Gateway Connect allows you to extend your SD-WAN into the AWS cloud without needing to set up any IPsec VPNs between your SD-Wan network virtual appliances and Transit Gateway.

This new type of attachment supports Generic Routing Encapsulation(GRE) which allows for higher bandwidth performance when compared to a standard VPN connection.

Transit Gateway connect supports BGP for your dynamic routing needs and removes the need to configure static routes.

The Connect Attachment itself is a new type of Transit Gateway attachment compared to what we have seen before - however, under the covers, the Connect attachment uses an existing VPC or AWS Direct Connect attachment as the underlying transport mechanism for your third party appliance to connect to.

Just for clarity, Your third party appliance is just a virtual router/gateway appliance running on an EC2 instance, in a connect VPC (a connect VPC is just the VPC that is running the appliance)

Take a look at this high-level example and you can see how a corporate data center and some other branches are able to connect into the connect VPC and are able to see all the adjacent VPCs attached to Transit Gateway.

If you want to learn more about this type of architecture - please check out this AWS blog talking all about it.

Some important service limits to be aware of:

Much like every other service with AWS, Transit Gateway has a number of Limits that you should be aware of:

In total we can create 5000 Transit Gateway attachments - those are the things that hook into your VPCs in each subnet, or attach to your VPN connections. Its a pretty large number so you are unlikely to ever run over this limit.

Speaking of VPN connections, you are also limited to 1.25Gbps per VPN tunnel. Keep in mind that each VPN connection does have two tunnels, so it’s not the worst. You can also use equal-cost multipath routing (ECMP) to get more bandwidth through your VPN by aggregating multiple VPN connections

The next important limit is the maximum bandwidth you can support within Transit Gateway is 50Gbps (at burst load)  per VPC, Direct Connect gateway, or peered Transit Gateway connection. So if you do find yourself having bandwidth issues - maybe its time to break up your architecture and connect up another VPC?

Another important limit to be aware of and, one that is fairly easy to hit - is the maximum number of Transit Gateways available. Which is five per region per account. If you do find yourself hitting this limit, again it might be time to think about separating our architecture into a multi-account structure.

And the last one I want to bring your attention to is there is a limit to the number of Transit Gateway attachments per VPC - and that number is five. So be careful of over-provisioning those attachment points.

There are more limits to the service, but I think the ones I just went over are the most important to be aware of.