Cisco made the process of site to site ipsec encrypted communications fairly easy with the introduction of virtual tunnel interfaces (VTI) in IOS version 12.2(13)T. The problems caused by the overhead of ipsec/ESP encapsulation of a payload are fairly well documented in their knowledge base document “Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC”. The “Readers Digest” version of the above article is that you need to reduce the IP mtu of the tunnel interface to a size that allows for the additional overhead of ipsec and/or GRE encapsulation.
Now as luck would have it I stumbled across a bug with tunnel interfaces miscalculating the IP mtu after the router is rebooted. For readers who just want the short story, the workaround is to always specify tunnel source by interface name, not ip address. Cisco TAC report that the bug exists across a large number of IOS versions and platforms. A bug ID has been requested from Cisco so that we can follow it. I’ll post it here once they allocate it. The bug ID is CSCth31172. For those that would like proof of the bug read on.
To test this I configured two 2811 routers in the lab. The configs can be viewed here for R1 and R2. The relevant interface configurations for each are below.
R1 config
interface Tunnel2 ip address 10.0.0.1 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 tunnel source Serial0/0/0.100 tunnel destination 192.168.0.2 tunnel mode ipsec ipv4 tunnel path-mtu-discovery tunnel protection ipsec profile MY_VTI ! interface Serial0/0/0 no ip address encapsulation frame-relay IETF clock rate 2000000 ! interface Serial0/0/0.100 point-to-point ip address 192.168.0.1 255.255.255.252 frame-relay interface-dlci 100
R2 Config
interface Tunnel1 ip address 10.0.0.2 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 tunnel source 192.168.0.2 tunnel destination 192.168.0.1 tunnel mode ipsec ipv4 tunnel protection ipsec profile MY_VTI ! interface Serial0/0/0 no ip address encapsulation frame-relay IETF frame-relay intf-type dce ! interface Serial0/0/0.100 point-to-point ip address 192.168.0.2 255.255.255.252 frame-relay interface-dlci 100
The serial interface on router R1 that carries the tunnel traffic should have an mtu of 1500 bytes. We verify this by doing a ping with the df bit set.
R1#ping 192.168.0.2 df size 1500 Type escape sequence to abort. Sending 5, 1500-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds: Packet sent with the DF bit set !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/16 ms
The IP mtu of the tunnel2 interface is configured to 1400 bytes, an allowance of 100 bytes for ESP header/trailer and GRE headers. That’s plenty, and we should be able to send 1400 bytes through with the DF bit set.
R1#ping 10.0.0.2 df size 1400 Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds: Packet sent with the DF bit set ..... Success rate is 0 percent (0/5)
Something is not quite right. We can check the IP mtu of the crypto SA by issuing the command.
R1#show crypto ipsec sa | include mtu|interface interface: Tunnel2 path mtu 1500, ip mtu 1500, ip mtu idb Serial0/0/0.100 interface: Tunnel21 path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
Nothing obvious there, time to move on to router R2 and repeat the tests. First confirm the mtu of the underlying interface.
R2#ping 192.168.0.1 df size 1500 Type escape sequence to abort. Sending 5, 1500-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds: Packet sent with the DF bit set !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/16 ms
All good, now we can test the tunnel to R1 with 1400 byte packets.
R2#ping 10.0.0.1 df size 1400 Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds: Packet sent with the DF bit set M.M.M *Jun 7 01:08:10.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343 *Jun 7 01:08:10.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343 *Jun 7 01:08:12.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343 *Jun 7 01:08:12.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343 *Jun 7 01:08:14.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343
Okay, so now we’re on to something. The crypto engine tells us the mtu is 1343 or 57 bytes short of our expectation. Coincidentally that is suspiciously close to the 52 bytes overhead for ESP that Cisco has documented in “QoS DESIGN FOR IPsec VPNs“.
R2#show crypto ipsec sa | include interface|mtu interface: Tunnel1 path mtu 1400, ip mtu 1400, ip mtu idb Tunnel1 interface: Tunnel12 path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
Notice the differences between tunnel 2 on R1 and Tunnel 1 on R2. The IP mtu is 1400 bytes and the idb (I have no idea what idb means Interface Descriptor Block, thanks for the correction Ivan) is the actual tunnel interface and not the transit interface of the tunnel. We can reset the tunnel interfaces and the crypto SA’s by briefly shutting the interface.
R2#conf t Enter configuration commands, one per line. End with CNTL/Z. R2(config)#int tu1 R2(config-if)#shut R2(config-if)#no shut *Jun 7 01:10:48.571: %LINK-5-CHANGED: Interface Tunnel1, changed state to administratively down *Jun 7 01:10:49.571: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to down *Jun 7 01:11:07.555: %LINK-3-UPDOWN: Interface Tunnel1, changed state to up *Jun 7 01:11:08.555: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up R2(config-if)#end *Jun 7 01:11:18.515: %SYS-5-CONFIG_I: Configured from console by console R2#show crypto ipsec sa | include interface|mtu interface: Tunnel1 path mtu 1500, ip mtu 1500, ip mtu idb Serial0/0/0.100 interface: Tunnel12 path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
The difference to tunnel1 after shutting down the interface is readily apparent. The IP mtu is now 1500 bytes and the idb is now the serial subinterface. A 1400 byte ping should now be possible.
R2#ping 10.0.0.1 df size 1400 Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds: Packet sent with the DF bit set !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms R2#
It’s not really viable resetting tunnel interfaces after every reboot, but our testing has found that Cisco’s workaround of specifying tunnel source by interface name rather than IP address has been 100% effective so far.
