The Moose and Squirrel Files

June 11, 2010

Cisco IPSEC MTU Bug

Filed under: Network — Tags: , , , — networknerd @ 9:52 am

Cisco made the process of site to site ipsec encrypted communications fairly easy with the introduction of virtual tunnel interfaces (VTI) in IOS version 12.2(13)T.  The problems caused by the overhead of ipsec/ESP encapsulation of a payload are fairly well documented in their knowledge base document “Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC”. The “Readers Digest” version of the above article is that you need to reduce the IP mtu of the tunnel interface to a size that allows for the additional overhead of ipsec and/or GRE encapsulation.

Now as luck would have it I stumbled across a bug with tunnel interfaces miscalculating the IP mtu after the router is rebooted.   For  readers who just want the short story, the workaround is to always specify tunnel source by interface name, not ip address. Cisco TAC report that the bug exists across a large number of IOS versions and platforms. A bug ID has been requested from Cisco so that we can follow it. I’ll post it here once they allocate it.  The bug ID is CSCth31172. For those that would like proof of the bug read on.

To test this I configured two 2811 routers in the lab.  The configs can be viewed here for R1 and R2.  The relevant interface configurations for each are below.
R1 config

interface Tunnel2
  ip address 10.0.0.1 255.255.255.252
  ip mtu 1400  ip tcp adjust-mss 1360
  tunnel source Serial0/0/0.100 
  tunnel destination 192.168.0.2
  tunnel mode ipsec ipv4
  tunnel path-mtu-discovery
  tunnel protection ipsec profile MY_VTI
!
interface Serial0/0/0
  no ip address
  encapsulation frame-relay IETF
  clock rate 2000000
!
interface Serial0/0/0.100 point-to-point
  ip address 192.168.0.1 255.255.255.252
  frame-relay interface-dlci 100

R2 Config

interface Tunnel1
  ip address 10.0.0.2 255.255.255.252
  ip mtu 1400
  ip tcp adjust-mss 1360
  tunnel source 192.168.0.2
  tunnel destination 192.168.0.1
  tunnel mode ipsec ipv4
  tunnel protection ipsec profile MY_VTI
!
interface Serial0/0/0
  no ip address
  encapsulation frame-relay IETF
  frame-relay intf-type dce
!
interface Serial0/0/0.100 point-to-point
  ip address 192.168.0.2 255.255.255.252
  frame-relay interface-dlci 100

The serial interface on router R1 that carries the tunnel traffic should have an mtu of 1500 bytes. We verify this by doing a ping with the df bit set.

R1#ping 192.168.0.2 df size 1500
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/16 ms

The IP  mtu of the tunnel2 interface is configured to 1400 bytes, an allowance of 100 bytes for ESP header/trailer and GRE headers.  That’s plenty, and we should be able to send 1400 bytes through with the DF bit set.

R1#ping 10.0.0.2 df size 1400

Type escape sequence to abort.
Sending 5, 1400-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
Packet sent with the DF bit set
.....
Success rate is 0 percent (0/5)

Something is not quite right. We can check the IP mtu of the crypto SA by issuing the command.

R1#show crypto ipsec sa | include mtu|interface
 interface: Tunnel2
      path mtu 1500, ip mtu 1500, ip mtu idb Serial0/0/0.100
 interface: Tunnel21
      path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0

Nothing obvious there, time to move on to router R2 and repeat the tests.  First confirm the mtu of the underlying interface.

R2#ping 192.168.0.1 df size 1500
 Type escape sequence to abort.
 Sending 5, 1500-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds: Packet sent with the DF bit set
 !!!!!
 Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/16 ms

All good, now we can test the tunnel to R1 with 1400 byte packets.

R2#ping 10.0.0.1 df size 1400
 Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds: Packet sent with the DF bit set
 M.M.M
*Jun  7 01:08:10.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343
*Jun  7 01:08:10.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343
*Jun  7 01:08:12.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343
*Jun  7 01:08:12.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343
*Jun  7 01:08:14.855: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1400, mtu=1343

Okay, so now we’re on to something.  The crypto engine tells us the mtu is 1343 or 57 bytes short of our expectation. Coincidentally that is suspiciously close to the 52 bytes overhead for ESP that Cisco has documented in “QoS DESIGN FOR IPsec VPNs“.

R2#show crypto ipsec sa | include interface|mtu
interface: Tunnel1
      path mtu 1400, ip mtu 1400, ip mtu idb Tunnel1
interface: Tunnel12
      path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0

Notice the differences between tunnel 2 on R1 and Tunnel 1 on R2.  The IP mtu is 1400 bytes and the idb (I have no idea what idb means Interface Descriptor Block, thanks for the correction Ivan) is the actual tunnel interface and not the transit interface of the tunnel. We can reset the tunnel interfaces and the crypto SA’s by briefly shutting the interface.

R2#conf t
 Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int tu1
R2(config-if)#shut
R2(config-if)#no shut
*Jun  7 01:10:48.571: %LINK-5-CHANGED: Interface Tunnel1, changed state to administratively down
*Jun  7 01:10:49.571: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to down
*Jun  7 01:11:07.555: %LINK-3-UPDOWN: Interface Tunnel1, changed state to up
*Jun  7 01:11:08.555: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up
R2(config-if)#end
*Jun  7 01:11:18.515: %SYS-5-CONFIG_I: Configured from console by console
R2#show crypto ipsec sa | include interface|mtu
interface: Tunnel1
      path mtu 1500, ip mtu 1500, ip mtu idb Serial0/0/0.100
interface: Tunnel12
      path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0

The difference to tunnel1 after shutting down the interface is readily apparent. The IP mtu is now 1500 bytes and the idb is now the serial subinterface.  A 1400 byte ping should now be possible.

R2#ping 10.0.0.1 df size 1400
Type escape sequence to abort.
Sending 5, 1400-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/14/16 ms
R2#

It’s not really viable resetting  tunnel  interfaces after every reboot, but our testing has found that Cisco’s workaround of specifying tunnel source by interface name rather than IP address has been 100% effective so far.

Advertisements

Create a free website or blog at WordPress.com.