Tunnel MTU problems often appear as selective failures: a ping works, a TCP handshake completes, and then a large transfer stalls. Encapsulation made the outer packet larger than the path can carry, while a firewall or endpoint prevented Path MTU Discovery from correcting the packet size.

The useful calculation starts with the smallest MTU on the outer path, subtracts every tunnel header to obtain the effective inner MTU, and then derives TCP MSS from the inner IP version. The examples below show the arithmetic and the places where a protocol or cloud provider requires a measured or documented value instead of a guessed constant.

Short answer: effective MTU equals outer-path MTU minus encapsulation overhead. For TCP without IP or TCP options, IPv4 MSS is effective MTU minus 40 bytes; IPv6 MSS is effective MTU minus 60 bytes. Preserve ICMP Packet Too Big signaling and treat MSS clamping as a TCP mitigation, not a replacement for correct MTU and PMTUD.

The tunnel MTU and MSS formulas

Keep four values separate:

  1. Outer-path MTU: the largest outer IP packet that the complete underlay path can carry without fragmentation.
  2. Tunnel overhead: outer IP, tunnel, security, transport, optional-field, and padding bytes added around an inner packet.
  3. Effective MTU: the largest inner IP packet that fits after encapsulation.
  4. TCP MSS: the TCP payload advertised for the inner path, excluding the fixed inner IP and TCP headers.
Effective MTU = outer-path MTU − tunnel overhead
IPv4 TCP MSS = effective MTU − 20-byte IPv4 header − 20-byte TCP header
IPv6 TCP MSS = effective MTU − 40-byte IPv6 header − 20-byte TCP header

RFC 6691 says the MSS value is based on the fixed IP and TCP headers. The sender accounts for variable IP and TCP options when constructing each segment. When an interface's effective MTU can vary, RFC 6691 recommends using its smallest effective MTU for the advertised MSS.

Header or fieldBytes used hereWhat changes it
Outer IPv4 header20IPv4 options add bytes.
Outer IPv6 header40IPv6 extension headers add bytes.
UDP header8Used by NAT-T and WireGuard data transport.
Base GRE header4Checksum, key, and sequence fields increase it.
Fixed inner TCP header20TCP options affect each packet, not the MSS formula's fixed-header subtraction.

Worked example 1: GRE over IPv4

A router carries a minimal GRE tunnel across a 1500-byte IPv4 underlay. RFC 2784's base GRE header is 4 bytes, and the outer IPv4 header is 20 bytes:

  1. Outer-path MTU: 1500
  2. Outer IPv4 plus base GRE: 20 + 4 = 24 bytes
  3. Effective inner MTU: 1500 − 24 = 1476
  4. Inner IPv4 TCP MSS: 1476 − 40 = 1436
  5. Inner IPv6 TCP MSS: 1476 − 60 = 1416

The result is 1476 MTU, 1436 IPv4 MSS, or 1416 IPv6 MSS. It applies only to the stated minimal GRE format. A GRE key, checksum, sequence number, IPsec protection, or another outer header changes the overhead. RFC 2890 defines optional GRE key and sequence-number fields; include the fields actually emitted by both tunnel endpoints.

Worked example 2: AWS AES-GCM IPsec with NAT-T

ESP does not have one universal overhead. RFC 4303 includes an ESP header, trailer, padding, integrity data, and optional fields whose sizes depend on mode and algorithms. A reproducible IPsec calculation therefore needs a named profile. AWS documents 1446 MTU for its AES-GCM-16 profile without NAT-T and 1438 MTU with NAT-T:

  1. Documented AES-GCM-16 MTU without NAT-T: 1446
  2. Documented AES-GCM-16 MTU with NAT-T: 1438
  3. NAT-T difference within the same profile: 1446 − 1438 = 8 bytes
  4. Inner IPv4 TCP MSS: 1438 − 40 = 1398
  5. Inner IPv6 TCP MSS: 1438 − 60 = 1378

The profile result is 1438 MTU, 1398 IPv4 MSS, or 1378 IPv6 MSS. The 8-byte comparison isolates NAT-T only because every other profile input stays fixed. Do not transfer these values to another provider or algorithm: confirm tunnel mode, outer IP version, NAT-T state, cipher, integrity algorithm, ESP padding, provider limits, and device documentation.

The CIDRtools link below deliberately opens its generic AES-CBC + SHA256 and NAT-T heuristic, which produces 1440 MTU and 1400 IPv4 MSS. That estimate is useful for exploring inputs, but it is not the AWS profile above and must not override a provider or device's documented values.

Compare the generic IPsec + NAT-T estimatePlan VPN transit addresses

Worked example 3: WireGuard over IPv4

WireGuard transports encrypted data in UDP. Its protocol description shows a data-packet header containing a 4-byte type/reserved field, 4-byte receiver index, and 8-byte counter. The AEAD ciphertext adds a 16-byte authentication tag, producing 32 bytes of WireGuard data-message overhead before outer UDP and IP headers.

For a 1500-byte IPv4 underlay and a 1440-byte, 16-byte-aligned inner packet:

  1. Outer IPv4: 20 bytes
  2. Outer UDP: 8 bytes
  3. WireGuard data fields and authentication tag: 32 bytes
  4. Total stated overhead: 20 + 8 + 32 = 60 bytes
  5. Effective inner MTU: 1500 − 60 = 1440
  6. Inner IPv4 TCP MSS: 1440 − 40 = 1400
  7. Inner IPv6 TCP MSS: 1440 − 60 = 1380

The result is 1440 MTU, 1400 IPv4 MSS, or 1380 IPv6 MSS for this exact example. WireGuard pads encrypted inner packets to a multiple of 16 bytes. An outer IPv6 header adds 20 bytes relative to outer IPv4, and different underlay MTUs or nested tunnels change the result. Treat a platform default such as 1420 as a starting point, not proof of the end-to-end PMTU.

Worked example 4: managed cloud VPN values

A managed gateway's documented limit overrides a generic ESP estimate because the provider controls part of the encapsulation and forwarding path. Use the provider's table for the selected algorithm and NAT traversal state.

AWS Site-to-Site VPN

AWS documents a maximum 1446-byte MTU and 1406-byte IPv4 MSS for AES-GCM-16 without NAT-T. With NAT-T enabled, the documented values become 1438 MTU and 1398 IPv4 MSS. For IPv6 carried inside IPv4, AWS lists 1386 and 1378 MSS respectively.

AWS exampleDocumented MTUIPv4 MSSIPv6-in-IPv4 MSS
AES-GCM-16, NAT-T disabled144614061386
AES-GCM-16, NAT-T enabled143813981378
AES-CBC + SHA1/SHA2-256, NAT-T enabled142213821362

The AES-GCM NAT-T row demonstrates the UDP cost directly:1446 − 1438 = 8 bytes, and 1406 − 1398 = 8 bytes. The AES-CBC result also shows why a single “IPsec overhead” number is unsafe across algorithms.

Azure VPN Gateway

Microsoft documents bidirectional TCP MSS clamping on Azure VPN Gateway. For traffic over the internet, its table lists 1360 bytes for IPv4 and 1340 bytes for IPv6. Those values differ by the 20-byte difference between fixed IPv4 and IPv6 headers and both correspond to a 1400-byte packet when only the fixed inner IP and TCP headers are added.

Use those gateway values for the Azure path instead of replacing them with the generic IPsec estimate above. Recheck the provider documentation when the gateway SKU, path type, or service changes; Microsoft's same table lists different values for ExpressRoute Gateway traffic.

MTU black-hole troubleshooting workflow

  1. Find the real outer path: include WAN links, PPPoE, VLAN stacking, provider edges, internet transit, and nested overlays. The physical interface's configured MTU may not be the end-to-end PMTU.
  2. Inventory emitted headers: record outer IP version, GRE options, UDP, NAT-T, ESP mode and algorithms, WireGuard, and every nested encapsulation.
  3. Start conservatively: use the smallest vendor-documented or calculated effective MTU and derive separate IPv4 and IPv6 MSS values.
  4. Preserve PMTUD: allow the required ICMP fragmentation-needed or Packet Too Big messages through firewalls and return paths. IPv6 routers do not fragment forwarded packets.
  5. Test more than ping: verify TCP transfers, UDP applications, IPv4, IPv6, both directions, and packet sizes near the proposed limit. A small echo request proves very little.
  6. Use MSS clamping narrowly: it can keep TCP SYN negotiation below a known path limit, but it does not resize UDP, ICMP, ESP, or already-established TCP flows.
  7. Observe and adjust: capture outer and inner packet sizes, watch ICMP errors and retransmissions, and keep a rollback value before raising MTU or MSS.

Common MTU and MSS mistakes

  • Subtracting overhead from 1500 without checking the path: a PPPoE or nested overlay underlay may already be smaller.
  • Counting only the named tunnel header: GRE also needs an outer IP header; WireGuard needs outer IP and UDP; NAT-T adds UDP around ESP.
  • Using IPv4 MSS for IPv6: the fixed IPv6 header is 20 bytes larger, so the same effective MTU produces an MSS 20 bytes smaller.
  • Treating ESP as fixed: algorithms, integrity data, mode, optional fields, and padding change the overhead.
  • Calling MSS clamping an MTU fix: it affects TCP negotiation only and can hide broken PMTUD while non-TCP traffic still fails.
  • Blocking all ICMP: this can create a PMTUD black hole where oversized packets disappear without actionable feedback.

Authoritative sources