Layer 3 Switching Fundamentals: How ASICs Enable High-Speed Inter-VLAN Routing and IP Forwarding in

Introduction: From Routers to Layer 3 Switches

Early networks primarily used Layer 2 switches to build local area networks (LANs), while communication between different LANs was handled by routers. In those days, traffic within a LAN constituted the vast majority of network traffic, and inter-network communication was relatively low, making a small number of routers sufficient. A typical network structure is shown in Figure 1.

Figure 1: Network using Layer 2 Switches and Routers

However, as data communication networks expanded and services diversified, the need for inter-network communication grew significantly. Routers, with their high cost, low forwarding performance, and limited number of ports, could not adequately meet the demands of this evolving network landscape. Since routers primarily use IP forwarding (Layer 3 forwarding) to interconnect different networks, a question arose: could the high performance of switches be applied to Layer 3 forwarding? The answer is yes, leading to the development of the Layer 3 switch, a device designed for high-speed Layer 3 forwarding.

Most Layer 3 switches use ASIC (Application-Specific Integrated Circuit) hardware chips to perform forwarding. These ASIC chips integrate the functions of IP forwarding, including:

• Checking the IP packet header.

• Modifying the Time-To-Live (TTL) parameter.

• Recalculating the IP header checksum.

• Performing data link encapsulation for the IP packet.

The network structure using a Layer 3 switch is shown in Figure 2.

Figure 2: Network using a Layer 3 Switch

Layer 3 Switch vs. Router: Key Differences

A router's Layer 3 forwarding relies primarily on its CPU, whereas a Layer 3 switch's forwarding is handled by ASIC chips. This fundamental difference results in a massive performance gap between the two.

It's important to note that Layer 3 switches cannot completely replace routers. Routers still excel in areas such as rich interface types, sophisticated Quality of Service (QoS) controls, and powerful routing capabilities, which are often weaknesses in Layer 3 switches.

Modern Layer 3 switches typically use VLANs to segment Layer 2 networks and perform Layer 2 switching, while also enabling Layer 3 IP communication between different VLANs.

Host Communication Behavior

Before discussing the forwarding principle of Layer 3 switches, it's essential to understand how hosts in different networks communicate:

1. Same Subnet: The source host first compares its own IP address with the destination host's IP address. If they are on the same subnet (determined by having the same network address after applying the subnet mask), the source host sends an ARP request directly to the destination host. Upon receiving the ARP reply, it obtains the destination's physical (MAC) address and uses it as the destination MAC address for sending packets. This occurs when hosts within the same VLAN (subnet) communicate, and the connecting switch performs Layer 2 switching.

2. Different Subnets: When the source host determines the destination is on a different subnet, it forwards the packet to its configured gateway. It sends an ARP request to obtain the MAC address corresponding to the gateway's IP address. After receiving the ARP reply from the gateway, it uses the gateway's MAC address as the destination MAC address for the packet. Note: The source IP in the packet is the host's IP, and the destination IP is still the target host's IP. This occurs when hosts in different VLANs (subnets) communicate, and the connecting switch performs Layer 3 switching.

Internal Structure of a Layer 3 Switch

To understand the forwarding principle, let's briefly examine the internal structure of a Layer 3 switch, as shown in Figure 3.

Figure 3: Layer 3 Switch Hardware Structure

The two main components are:

1. ASIC: Handles the primary Layer 2 and Layer 3 forwarding functions. It contains a MAC address table for Layer 2 forwarding and a Layer 3 forwarding table for IP routing.

2. CPU: Handles control plane functions. It maintains software tables (including the software routing table and software ARP table) and configures the ASIC's hardware Layer 3 forwarding table based on the information in these software tables. The CPU can also perform software-based Layer 3 forwarding itself.

The high-speed switching performance is determined by the hardware tables within the ASIC, which are populated based on the software tables maintained by the CPU.

Layer 3 Forwarding Process: Scenario 1 (Hosts on the Same Switch)

Let's explain the forwarding principle using a scenario where two hosts connected to the same Layer 3 switch but in different VLANs need to communicate, as shown in Figure 4.

Figure 4: Layer 3 Forwarding - Scenario 1

Assume the Layer 3 switch has no pre-existing hardware entries. When PC A sends an ICMP request to PC B:

1. PC A determines that the destination IP (2.1.1.2) is on a different subnet. It sends an ARP request to resolve the MAC address of its gateway (1.1.1.1).

2. The Layer 3 switch (L3_SW) receives the ARP request, recognizes that the requested IP is its own interface IP, and sends an ARP reply containing its MAC address (MAC S). It also records PC A's IP-to-MAC mapping (1.1.1.2 <> MAC A) in its ARP table.

3. PC A receives the gateway's ARP reply and constructs an ICMP request packet with: Destination MAC = MAC S, Source MAC = MAC A, Source IP = 1.1.1.2, Destination IP = 2.1.1.2.

4. L3_SW receives the packet. It first updates its MAC address table with the source MAC and VLAN ID (VID). Then, it looks up the destination MAC + VID in its MAC table, finding an entry matching its own Layer 3 interface MAC (which has a "Layer 3 forward" flag set). This triggers a lookup in the ASIC's Layer 3 table.

5. The chip looks up the destination IP in the Layer 3 table. Finding no entry (first packet), it sends the packet to the CPU for software processing.

6. The CPU checks its software routing table, finds a matching direct route for PC B's subnet, then checks its ARP table for PC B's MAC (fails). L3_SW then broadcasts an ARP request for 2.1.1.2 on VLAN 3.

7. PC B replies with its MAC address (MAC B). L3_SW records this IP-to-MAC mapping (2.1.1.2 <> MAC B) in its ARP table.

8. L3_SW forwards the original ICMP request to PC B, changing the destination MAC to MAC B and the source MAC to its own MAC S. Simultaneously, it creates a new entry in the ASIC's Layer 3 table for PC B's IP, including the next-hop MAC, egress VLAN, and port. Subsequent packets from PC A to PC B will be hardware-switched using this entry.

9. PC B sends an ICMP reply to PC A. Since L3_SW already has the necessary ARP and Layer 3 entries for PC A, this reply is forwarded directly by the ASIC hardware.

This process illustrates the core principle of "Route Once, Switch Many." The first packet is routed by the CPU to establish the hardware forwarding path, and all subsequent packets are switched at wire speed by the ASIC.

Layer 3 Forwarding Process: Scenario 2 (Hosts on Different Switches)

Now, let's consider a more complex scenario where two hosts are connected to different Layer 3 switches, as shown in Figure 5.

Figure 5: Layer 3 Forwarding - Scenario 2

Assume static routes are configured on both switches. The process is similar to Scenario 1.

1. PC A sends its packet to its gateway, L3_SW1 (using MAC S1).

2. L3_SW1 receives the packet. The ASIC finds no Layer 3 entry and sends it to the CPU.

3. The CPU on L3_SW1 finds a route for PC B's network with a next-hop of 3.1.1.2 (L3_SW2). If it doesn't have an ARP entry for 3.1.1.2, it performs an ARP request.

4. After getting the ARP reply (3.1.1.2 <> MAC S2), L3_SW1 forwards the packet to L3_SW2, changing the destination MAC to MAC S2 and the source MAC to its own MAC S1. It also installs a hardware Layer 3 entry for the 2.1.1.0/24 network in its ASIC.

5. L3_SW2 receives the packet. The process repeats similarly to Scenario 1 inside L3_SW2: it eventually ARPs for PC B, then forwards the packet, changing the destination MAC to MAC B and the source MAC to its own MAC S2. It also installs a hardware entry.

6. PC B's reply follows the reverse path, now leveraging the hardware entries installed on both L3_SW2 and L3_SW1.

Characteristics of Layer 3 Switching

From the above processes, we can summarize the key characteristics:

1. First packet is routed by the CPU to establish the hardware flow; subsequent packets are switched by the ASIC ("Route Once, Switch Many").

2. The ASIC's hardware forwarding focuses on the information in its table (destination IP/MAC, egress VLAN), not the abstract next-hop IP from the routing table.

3. Each time a packet undergoes Layer 3 forwarding, its source and destination MAC addresses change, while the source and destination IP addresses remain unchanged.

ASIC Implementation: Matching Algorithms

The performance and characteristics of a Layer 3 switch depend heavily on its ASIC implementation, particularly the structure and lookup method of the hardware Layer 3 table. There are two primary matching algorithms:

1. Exact Match (Host Routes): Early ASICs stored only exact IP addresses (like 2.1.1.2) without mask information. This required a separate hardware entry for every destination IP address, which was inefficient and vulnerable to resource exhaustion from network scans (e.g., during the Blaster worm outbreak).

2. Longest Prefix Match (LPM - Network Routes): Modern ASICs store IP addresses with subnet masks (e.g., 2.1.1.0/24). Lookups follow the longest prefix match rule. Routes are often pre-loaded into the hardware.

   • For a direct route, the initial entry is flagged "to CPU." The first packet triggers an ARP request, and the learned host route (/32) is installed as a more specific entry for hardware forwarding.

   • For an indirect route, the entry points to the next-hop. If the next-hop's MAC is unknown, the entry is flagged "to CPU." The first packet triggers an ARP for the next-hop, and the entry is updated with the MAC and unflagged, allowing hardware forwarding for all subsequent packets matching that route.

Most modern Layer 3 switches support both exact-match entries (for host ARP entries) and longest-prefix-match entries (for network routes).

ASIC Implementation: Table Structure

The structure of the hardware Layer 3 table also varies, primarily regarding the inclusion of the egress port:

• Table Includes Egress Port: The lookup yields the output port directly. This is simple and fast but requires the CPU to update the hardware table if the physical topology changes (e.g., due to STP or link aggregation), which can be complex and error-prone.

• Table Excludes Egress Port: The lookup yields only the next-hop MAC and egress VLAN. A subsequent MAC address table lookup is then performed to determine the egress port. This adds a step but is more robust, as topology changes are automatically reflected in the MAC table without needing to update the Layer 3 table.

Different switch manufacturers choose different implementations based on cost, reliability, and product positioning.