Routing & Connectivity in AWS, Part 1 - A Guide for Network Engineers

Intro

In our previous posts of this course, we laid out the foundation for AWS networking by discussing VPC fundamentals and base cloud connectivity concepts. Now, we turn the page to Routing and Connectivity in AWS.

If you're a network engineer coming from traditional data centre or campus networking, many of the principles/ concepts will sound familiar. We still deal with precious IP subnets, routing propagation and gateways, but the implementation in AWS differs in many ways. But don't shy away just yet. In my opinion, AWS networking is significantly easier to get to grips with and understand compared to traditional networking, and having experience as a Network Engineer will greatly improve your chances of one-shotting your knowledge grasp around these differences and topics.

In this chapter of our series, we'll run through technologies and services such as;

Part 1

• AWS Routing - Key Differences
• VPC Peering
• Transit Gateway
• AWS PrivateLink & VPC Endpoints

Part 2

• BGP (in AWS)
• Direct Connect
• VPN Connections
• and much, much more..

The goal here is to give you a clear understanding of how these AWS connectivity options work, and how they compare to the tools and techniques you've potentially mastered in physical networks.

Key Differences in AWS Routing

Before we begin with the meat and bones of this post, I wanted to run through a high-level glance at the key differences in AWS routing vs a traditional network.

In a traditional network you may configure your routers (network) with OSPF and/ or BGP to dynamically share routes. In AWS (the new world), the platform handles and offloads much of the routing itself and, as such, each subnet we create has an associated route table where you define static routes for traffic leaving that subnet.

There isn't a requirement to run an IGP (Interior Gateway Protocol) like OSPF inside the VPC; AWS's underlying infrastructure ensures packets reach their destination within a VPC.

On the flip side, when connecting across networks - such as between VPCs or between AWS and your on-premises environment, you will encounter familiar protocols like ye olde faithful BGP (Border Gateway Protocol) to exchange routes dynamically.

BGP and AWS

What is a routing protocol? A routing protocol is a mechanism that enables the exchange and distribution of routing information between routers, so routing decisions can be made and paths created. While you may be familiar with IP addresses, without routing protocols, we can’t just say we want to reach,

Art of InfraDan Jones

In a nutshell, inside a VPC, you control routing with managed route tables (static routes), and at the edges of your AWS environment, AWS allows dynamic routing (primarily via BGP) to integrate with external networks.

Keep this context in mind as we delve into the AWS connectivity options and solutions.

VPC Peering - Linking Two Virtual Networks

What is VPC Peering? VPC Peering is the most delightfully-straightforward approach to connecting two VPCs, so they can communicate as if they're on the same network.

VPC Peering is a direct, one-to-one network link between two VPCs, enabling routing of private IP traffic between them. VPC Peering connections can work across accounts and even across regions. Traffic over an inter-regional peer stays on the AWS global backbone (encrypted in transit), ensuring your precious packets don't touch the public internet.

Once a peering connection has been established and routes to direct the traffic from one VPC to another implemented, Instances in one VPC can communicate with instances in the other peered VPC. Subject to your Security Groups & NACLs allowing to do so, and not in total shambles.

VPC Peering Use

After creating a peering connection (and accepting it on the other side), you must update the route tables in each VPC to direct traffic for the other VPC’s CIDR range over the peer. You’ll also need to ensure Security Groups and NACLs allow the desired traffic. Once that’s done, the two VPCs can talk!

Network engineers will find this similar to configuring a static route between two routers for a directly connected network. This makes it extremely straightforward, but only suitable for a small number of connections in a limited number of use-cases.

Considerations and Limits

VPC Peering is non-transitive. What does this mean? Say there were three VPCs connected, daisy-chained together (VPC1 > VPC2 > VPC3), VPC1 can communicate with VPC2, but not VPC3, and vice versa. But VPC2 can communicate with VPC1 and VPC3.

If a VPC is not directly peered with another, they cannot communicate over VPC Peering Connections.

Each pair of VPCs that need to send traffic to one-another, requires its own peering connection and route table updates. You can probably imagine that as the number of VPCs grows, managing a full mesh of peering (with each VPC requiring peers to all others) becomes complex and unmaintainable.

As a separate note, there is a limit of 125 of these peering connections per VPC, so at large scale, peering alone just isn't workable. Additionally, VPC CIDR overlap is not allowed within peers, meaning the two VPCs' primary IPv4 ranges must not conflict.

Performance? VPC peering connections have no user-configurable bandwidth setting (much the same in AWS in general), instead, they utilise the full available bandwidth between instances.

Within AWS, there is a 5Gbps per TCP/ UDP flow limit, however, multiple flows (like utilising MPTCP) can maximise throughput. This makes peering suitable for high-bandwidth, low-latency requirements when directly connecting two VPCs, with the only trade-off being the lack of centralised management and the transitive routing limitation.

The transitive routing limitations do not prevent you from being creative, however. We could implement a proxy inside VPC 4 for example, which would allow internet connectivity through an intermediary VPC. But even though we could in theory get creative, it doesn't mean we should, as adding complexity adds to technical debt.

Use Cases 

• Great for simple, pair-wise VPC integrations.
  • Example: Connecting a corporate services VPC to a business units' VPC for limited sharing.
  • Example: Great for configuring a link and connecting to a partner or third-party VPC to access specific resources (B2B data sharing, resource sharing/ access)
• Quick and effective connection of two VPCs.
• High-speed interoperability between 2 VPCs.

As soon as you need a hub-and-spoke or many-to-many network topology, or you anticipate adding many more VPCs, you’ll quickly run into the management burden of peering. This is where AWS Transit Gateway comes into play.

AWS Transit Gateway - A Central Routing Hub

AWS Transit Gateway is the cloud router service provided by AWS. If you're from a network engineering background, think of TGW as a scalable virtual router that sits in the middle of your network topology and has interfaces (TGW attachments) to each of your networks.

Instead of creating dozens of peering links, like we saw earlier, you can attach all your VPCs (and other network connections) to a Transit Gateway, and it will route traffic between them based on its routing tables.

The best way to describe it is; Transit Gateway acts as a central hub that controls how traffic is routed among all the connected networks (the spoke VPCs and other connections).

VPC Peering Connections vs Transit Gateway

You may have encountered the terms “VPC Peering Connection” & “Transit Gateway” during your time as an engineer working on infrastructure in AWS, but what do they mean, and where do they differ? Hopefully, after this post, you’ll be able to distinguish the difference between the two immediately and when to

Art of InfraDan Jones

This hub-and-spoke model greatly simplifies network design and connectivity as your cloud footprint grows, which you'll absolutely crave following your 10th VPC peering connection. 

How Transit Gateway works

You create a TGW in a region and then attach networks to it. Attachments can be of several types:

• VPC attachments
  • To connect a single VPC
• VPN attachments
  • To connect an external network via Site-to-Site VPN
• Direct Connect (DX) attachments
  • To connect your premises directly to the AWS network. More on this later.
• Transit Gateway Peering attachments
  • To connect TGWs in different regions
• Transit Gateway Connect
  • For integrating SD-WAN appliances

the Aoi discord community

Join our new Discord community where we discuss everything; infra, cloud, AI and much more. Come vent, discuss tech problems, or just have a place to hang out with like-minded individuals.

Join Now

The TGW itself has its own routing tables (default or multiple route tables to segment routing domains). For each attachment, you can specify which TGW route table it uses. Routes in the TGW route table direct traffic to an attachment (next-hop), similar to how a router forwarding table directs packets out the correct interface.

Transit Gateway doesn't just support static routes; it also provides capabilities for route propagation. For example, when you attach a VPC, the TGW can automatically propagate that VPC's subnet CIDR into the TGW.

Transit Gateway supports both static routes and route propagation. For example, when you attach a VPC, the TGW can automatically propagate that VPC’s subnet CIDR into the TGW route table. Likewise, routes learned via BGP (external connections) can be propagated into TGW route tables dynamically. This means less manual administration (keyboard bashing) of routes compared to managing dozens of VPC peering links and route tables individually.

Scale and Performance

Transit Gateways are designed for scale. A single TGW can have from one to 5000 attachments (other networks or VPCs). Bandwidth is also high, each VPC attachment to a TGW can support up to 100Gbps of traffic burst per availability zone. So if your VPC spans multiple AZs (Availability zones), each AZ gets its own 100Gbps attachment to the TGW, effectively partitioning traffic by AZ for high throughput.

Compared to what VPC peering or individual VPN attachments can handle, these limits are beyond orders of magnitude, making TGW suited for large enterprise networks in AWS. There are other soft limits, however, to be mindful of;

• 10,000 combined routes across the 20 route tables TGW supports.
  • These routes, however, can be summarised where possible. To increase these limits, a support call can be logged with AWS.
• 7,500,000 packets per second per VPC AZ > TGW attachment
• Max MTU (maximum transmission unit) of 8500 bytes per VPC, AWS Direct Connect, Transit Gateway Connect, and peering attachment.
  • 1500 bytes per VPN attachment/ connection.

Use Cases 

• Hub-and-spoke cloud networks
  • Multiple VPCs connecting through a central hub.
• Hybrid connectivity
  • Where on-premises networks come in via a VPN or Direct Connect and require access to (many) VPCs.
• Segmented architecture
  • Where you use multiple route tables on the TGW to isolate certain traffic flows (production, dev, etc)
• Central egress or security architectures.
  • Where you require a single point to egress traffic out/ in over the internet or a central inspection/ security VPC for scanning all traffic before being handed back to the TGW to route.
• Inter-region peering.
  • Allowing you to connect TGWs in different AWS regions to build a global network (though AWS Cloud WAN, discussed later, is a shiny new service for globally unified networks).

Cost and Management

Transit Gateway is a managed service; as such, it incurs additional cost.

• $0.05 per hour, per attachment. (US East, 2025, at time of writing)
• £0.02 per GB of traffic/ data processed. (US East, 2025, at time of writing)

As a network engineer working in the cloud arena, you should plan for this in the design. For large volumes of east-west traffic, these costs can mount up quickly, but the trade-off if often worth it considering the reduction in complexity (and tech debt) and improved future-proofing.

As noted earlier, and in a previous post, if you foresee expansion (more VPCs, more hybrid links), starting with a Transit Gateway can save you from re-architecting later.

In summary, Transit Gateway is to AWS VPCs what a core router is to a campus network, a central point to connect and route between multiple networks, with the ability to enforce segmentation and use dynamic routing. It’s a primary tool in the AWS networking toolkit for building scalable, manageable cloud networks.

Key Takeaway Points for VPC Peering vs. Transit Gateway

• VPC Peering is a one-to-one private link between two VPCs. It’s simple and performant for connecting a pair of VPCs, but it does not scale well for many VPCs (no transitive routing and a limit of 125 peers). Use it for small-scale or specific VPC-to-VPC connectivity needs.
• AWS Transit Gateway is a hub-and-spoke connectivity model. It can connect hundreds or thousands of networks through a central router, making it ideal for enterprise-scale AWS environments. It simplifies route management (one central hub versus many peering links) and supports dynamic route propagation. Use TGW when you need a scalable, future-proof network architecture in AWS.

AWS PrivateLink and VPC Endpoints - Service Connectivity

So far, we've tackled VPC-to-VPC connectivity at the network level. So let's take a bit of a detour and discuss another form of connectivity that is service-level; AWS VPC Endpoints and PrivateLink.

VPC Endpoints and PrivateLink allow you to connect to AWS services, or to services in another VPC without connecting the networks via peering or TGW. For a network engineer, this is similar to exposing a service via a private, direct link/ dark fibre rather than opening up network routes.

VPC and Gateway Endpoints

An Interface Endpoint is essentially an elastic network interface (ENI) that gets deployed into your subnet, providing a private IP address entry point to a supported AWS service powered by AWS PrivateLink.

When you create an interface endpoint for, say, Amazon S3, AWS will set up that ENI in your VPC. Your instances inside that VPC can then resolve a domain for the service (which DNS will direct to the endpoint's private IP) and communicate with the service privately, within AWS's network. This traffic never goes out to the internet; it stays within AWS. So, there are no requirements for Internet Gateways.

Interface endpoints can also be used to reach endpoint services (resources) offered by other VPCs or Accounts, including third-party SaaS providers or even your own VPC in another account. When utilising interface endpoints in situations such as these, the consumer only sees a private IP in their own VPC, like a local service, and doesn't need to route to the provider's VPC at all. This avoids issues like overlapping IP ranges or managing cross-network routing.

These examples are a one-way connection: consumers initiate traffic to the service endpoint, but the service VPC cannot arbitrarily send traffic back into the consumer VPC. This unidirectional nature is a security feature (in contrast with VPC peering, which is bidirectional by default).

A Gateway Endpoint is a different type of VPC endpoint, specifically for accessing S3 or AWS DynamoDB. Instead of creating a network interface, a gateway endpoint is like a route target that you attach to your VPC. Simply update your route table to direct S3 or DynamoDB traffic to this endpoint. It's conceptually similar to having a virtual route to S3/ DynamoDB within the AWS network.

Gateway endpoints do not use PrivateLink, nor do they require Internet Gateways, and are not ENIs; they’re simply an invisible gateway for those services. They are free to use and recommended for S3/Dynamo access from a VPC without internet. The downside is they only work for those two services and cannot be shared across VPCs. Only resources in the same VPC can use the gateway endpoint.

When to use PrivateLink (Interface/ Gateway Endpoints)

PrivateLink (for interface endpoints) is incredibly useful when you want to expose or consume a service without exposing entire networks. Some common scenarios below:

• Shared services VPC (Or partner/ SaaS provider VPC)
  • Accessing an API or application, or other VPCs to consume it.
  • No concern for IP overlap (can have the same CIDR ranges either end, it's no issue!).
  • Simple to set up, with consumers/ providers not requiring any knowledge of where the service VPC is.
  • No internet access required (IGW - Internet Gateway)
  • Fantastic for multi-account architecture or third-party services.
• Access AWS public services privately
  • S3, SNS, etc support interface endpoints.
  • EC2 and other compute resources can access these within private subnets without a need for an IGW or NAT Gateway
  • Improves security posture(no traversal of the public internet)
  • Reduction in latency and cost.
    • Example: Accessing S3 via a VPC endpoint keeps traffic on the AWS backbone and avoids public data transfer charges.
• Strict security requirements
  • Security architecture/ posture prohibits internet connectivity/ traversal.
  • PrivateLink endpoints enable service access in highly locked-down environments, such as those with sensitive data.

From the perspective of a network engineer, PrivateLink is like provisioning an MPLS VPC route to a specific service, rather than bridging entire networks. It simplifies connectivity by abstracting it at the service level.

It also decouples IP addressing; the two sides of PrivateLink can use overlapping IP ranges with no issues, since the consumer only talks to the endpoint IP (not the service VPC IP addresses.

PrivateLink vs. VPC Peering Summary

So, what are the core differences here with its nearest sibling, VPC Peering?

• PrivateLink allows one-way access to a specific service (app/ endpoint) in another VPC, via a private endpoint interface.
• VPC peering allows full two-way network connectivity between all resources in two VPCs.

If you trust the two interconnecting environments and want full integration, VPC peering or TGW is better. If you want to share only a particular capability or service, PrivateLink is ideal. Also, PrivateLink is perfect for cross-account situations where you might not want to or be able to peer networks.

On the flip side, PrivateLink requires the service to be fronted by a Network Load Balancer (discussed in a future chapter) and supports TCP/UDP for interface endpoints (it may not work for all types of protocols easily, since its connection-oriented). So it’s most suitable for things like HTTP APIs, databases, or other TCP services.

Cost and Management

• Each interface endpoint you create incurs an hourly cost, and the data processed through it also incurs a separate fee.
  • Endpoint hour cost: $0.01 (at the time of writing, 2025)
  • Endpoint data processing cost: Starting at $0.01 per GB

In design, consider placing endpoints only in the VPCs where needed. You can also attach an endpoint policy to restrict what resource the endpoint can access (for AWS services), and by setting up Security Groups on the endpoint ENI, you can control which instances can talk to it, giving fine-grained control that all network engineers can appreciate.

In summary, AWS PrivateLink and VPC Endpoints are powerful tools to simplify network architecture by keeping traffic private and avoiding the need for full network connectivity for every use case. They complement the broader connectivity options: use PrivateLink when you just need service access across boundaries, and use peering or TGW when you truly need network-to-network connectivity.

Recap

Below, we'll run through some of the key points gone through in this post.

• Static Routes in AWS: In AWS, each subnet has an associated route table where static routes are defined for traffic leaving that subnet.
• Routing Protocols in AWS: AWS uses BGP for dynamic routing to integrate with external networks, while managed route tables handle routing within a VPC.

• VPC Peering: VPC peering creates a direct network link between two VPCs, enabling private IP traffic routing and communication as if they were on the same network.
  • VPC Peering Basics: A one-to-one private link between two VPCs, suitable for small-scale or specific VPC-to-VPC connectivity needs.
  • VPC Peering Benefits: VPC peering allows instances in different VPCs to communicate, supports cross-account and cross-region connections, and utilises the existing AWS infrastructure without requiring additional appliances.
  • VPC Peering Functionality: Allows communication between VPCs by updating route tables and configuring security groups/NACLs.
  • VPC Peering Limitations: Non-transitive, requiring individual peering connections for each VPC pair, limited to 125 per VPC, and CIDR overlap is not allowed. Standard AWS 5Gbps per flow limit, but can be overridden utilising MPTCP, etc.
• VPC Peering Scalability: Becomes complex and unmaintainable at large scale because of the need for a full mesh of peering connections.
• VPC Peering Use Cases: Simple, pairwise VPC integrations, connecting to a partner or third-party VPC.

• Transit Gateway (TGW): A central hub for connecting multiple VPCs, VPNs, Direct Connect links, and other Transit Gateways across different AWS regions.
  • TGW Basics: A hub-and-spoke connectivity model, ideal for enterprise-scale AWS environments with scalable and future-proof network architecture.
  • Transit Gateway Advantages: Centralised routing hub for managing multiple VPCs, scalable virtual router for complex network topologies enabling instant connectivity between attached networks.
  • Transit Gateway Functionality: Connects VPCs, on-premises networks, and AWS regions, enabling traffic segmentation and central ingress/egress points.
  • TGW Attachment Types: VPC, VPN, Direct Connect, Transit Gateway Peering, and Transit Gateway Connect.
  • TGW Routing: Uses route tables to direct traffic between attachments, with support for static routes and route propagation.
  • Transit Gateway Functionality: Supports static routes and route propagation, reducing manual administration compared to managing VPC peering links and route tables individually.
  • Scalability and Performance: Designed for large enterprise networks, allowing up to 5000 attachments with high bandwidth per availability zone (up to 100Gbps).
• Transit Gateway Cost: Includes hourly attachment fees and data processing charges, which can be significant for large volumes of east-west traffic.
• Transit Gateway Benefits: Simplifies network design, reduces complexity, and future-proofs infrastructure for scalability and management.

• AWS PrivateLink and VPC Endpoints: Service-level connectivity options that allow connection to AWS services or services in another VPC without network peering.
  • Interface Endpoint Definition: An elastic network interface (ENI) providing a private IP address entry point to a supported AWS service.
  • Interface Endpoint Functionality: Allows instances within a VPC to communicate with AWS services privately, without going through the internet.
  • Interface Endpoint Usage: Enables access to endpoint services in other VPCs, accounts, or third-party SaaS providers, offering a secure, unidirectional connection.
  • Gateway Endpoint Functionality: Provides access to S3 or DynamoDB within a VPC without requiring internet access.
  • Gateway Endpoint Limitations: Only works for S3 and DynamoDB, cannot be shared across VPCs, and only accessible by resources within the same VPC.
  • PrivateLink Use Cases: Useful for exposing or consuming services without exposing entire networks, such as shared services VPCs, API access, or scenarios with potential IP overlap. Multi-account architectures, third-party services, and highly secured environments.
  • PrivateLink Benefits: Improves security, reduces latency and cost, and simplifies connectivity.
  • PrivateLink vs. VPC Peering: PrivateLink provides one-way access to a specific service, while VPC peering allows full two-way connectivity between VPCs.
  • PrivateLink Cost: Hourly cost for each interface endpoint and data processing fee.
  • PrivateLink Management: Endpoint policies and Security Groups provide fine-grained control over resource access and network communication.

Thank You

Thank you for taking the time to read this and I hope it was of some help on your AWS networking journey.

Please consider commenting below with your thoughts around the topics discussed, anything more you feel should be detailed, or just to say Thanks!.

Next Chapter - Coming soon

• Hybrid Connectivity
• Site-to-Site VPNs
• Direct Connect
• BGP in AWS

Chapter 1 - The Building Blocks of Cloud Networking

Chapter 2 - VPCs - A Guide for Network Engineers

Chapter 3 - Routing and Connectivity in AWS - Part 1

Chapter 4 - Routing and Connectivity in AWS - Part 2 (Coming Soon)