IPsec VPN troubleshooting

A VPN connection has multiple stages that can be confirmed to ensure the connection is working properly. It is easiest to see if the final stage is successful first since if it is successful the other stages will be working properly. Otherwise, you will need to work back through the stages to see where the problem is located.

When a VPN connection is properly established, traffic will flow from one end to the other as if both ends were physically in the same place. If you can determine the connection is working properly then any problems are likely problems with your applications.

On some FortiGate units, such as the FortiGate 94D, you cannot ping over the IPsec tunnel without first setting a source-IP. In this scenario, you must assign an IP address to the virtual IPSEC VPN interface. Anything sourced from the FortiGate going over the VPN will use this IP address.

If the egress/outgoing interface (determined by kernel route) has an IP address, then use the IP address of the egress/outgoing interface. Otherwise, use the IP address of the first interface from the interface list (that has an IP address).

The first diagnostic command worth running, in any IPsec VPN troubleshooting situation, is the following:

diagnose vpn tunnel list

This command is very useful for gathering statistical data such as the number of packets encrypted versus decrypted, the number of bytes sent versus received, the SPI identifier, etc. This kind of information in the resulting output can make all the difference in determining the issue with the VPN.

Another appropriate diagnostic command worth trying is:

diagnose debug flow

This command will inform you of any lack of firewall policy, lack of forwarding route, and of policy ordering issues.

The following is a list of such potential issues. Bear in mind that the troubleshooting suggestions below are not exhaustive, and may not reflect your network topology.

The options to configure policy-based IPsec VPN are unavailable.

Go to System > Feature Select. Select Show More and turn on Policy-based IPsec VPN.

If your VPN fails to connect, check the following:

• Ensure that the pre-shared keys match exactly (see The pre-shared key does not match (PSK mismatch error) below).
• Ensure that both ends use the same P1 and P2 proposal settings (see The SA proposals do not match (SA proposal mismatch) below).
• Ensure that you have allowed inbound and outbound traffic for all necessary network services, especially if services such as DNS or DHCP are having problems.
• Check that a static route has been configured properly to allow routing of VPN traffic.
• Ensure that your FortiGate unit is in NAT/Route mode, rather than Transparent.
• Check your NAT settings, enabling NAT traversal in the Phase 1 configuration while disabling NAT in the security policy. You might need to pin the PAT/NAT session table, or use some of kind of NAT-T keepalive to avoid the expiration of your PAT/NAT translation.
• Ensure that both ends of the VPN tunnel are using Main mode, unless multiple dial-up tunnels are being used.
• If you have multiple dial-up IPsec VPNs, ensure that the Peer ID is configured properly on the
• FortiGate and that clients have specified the correct Local ID.
• If you are using FortiClient, ensure that your version is compatible with the FortiGate firmware by reading the FortiOS Release Notes.
• If you are using Perfect Forward Secrecy (PFS), ensure that it is used on both peers. You can use the diagnose vpn tunnel list command to troubleshoot this.
• Ensure that the Quick Mode selectors are correctly configured. If part of the setup currently uses firewall addresses or address groups, try changing it to either specify the IP addresses or use an expanded address range. This is especially useful if the remote endpoint is not a FortiGate device.
• If XAUTH is enabled, ensure that the settings are the same for both ends, and that the FortiGate unit is set to Enable as Server.
• Check IPsec VPN Maximum Transmission Unit (MTU) size. A 1500 byte MTU is going to exceed the overhead of the ESP-header, including the additional ip_header,etc. You can use the diagnose vpn tunnel list command to troubleshoot this.
• If your FortiGate unit is behind a NAT device, such as a router, configure port forwarding for UDP ports 500 and 4500.
• Remove any Phase 1 or Phase 2 configurations that are not in use. If a duplicate instance of the VPN tunnel appears on the IPsec Monitor, reboot your FortiGate unit to try and clear the entry.

If you are still unable to connect to the VPN tunnel, run the following diagnostic command in the CLI:

diagnose debug application ike -1
diagnose debug enable

The resulting output may indicate where the problem is occurring. When you are finished, disable the diagnostics by using the following command:

diagnose debug reset
diagnose debug disable

The VPN tunnel goes down frequently.

If your VPN tunnel goes down often, check the Phase 2 settings and either increase the Keylife value or enable Autokey Keep Alive.

The pre-shared key does not match (PSK mismatch error).

It is possible to identify a PSK mismatch using the following combination of CLI commands:

diag vpn ike log filter name <phase1-name> 
diag debug app ike -1
diag debug enable

This will provide you with clues as to any PSK or other proposal issues. If it is a PSK mismatch, you should see something similar to the following output:

ike 0:TRX:322: PSK auth failed: probable pre-shared key mismatch
ike Negotiate SA Error:

The SA proposals do not match (SA proposal mismatch).

The most common problem with IPsec VPN tunnels is a mismatch between the proposals offered between each party. Without a match and proposal agreement, Phase 1 can never establish. Use the following command to show the proposals presented by both parties.

diag debug app ike -1
diag debug enable

The resulting output should include something similar to the following, where blue represents the remote VPN device, and green represents the local FortiGate.

responder received SA_INIT msg
incoming proposal:
proposal id = 1:
   protocol = IKEv2:
      encapsulation = IKEv2/none
      type=ENCR, val=AES_CBC (key_len = 256)
      type=INTEGR, val=AUTH_HMAC_SHA_96
      type=PRF, val=PRF_HMAC_SHA
      type=DH_GROUP, val=1536.
proposal id = 2:
   protocol = IKEv2:
      encapsulation = IKEv2/none
      type=ENCR, val=3DES_CBC
      type=INTEGR, val=AUTH_HMAC_SHA_2_256_128
      type=PRF, val=PRF_HMAC_SHA2_256
      type=DH_GROUP, val=1536.
proposal id = 1:
   protocol = IKEv2:
      encapsulation = IKEv2/none
      type=ENCR, val=AES_CBC (key_len = 128)
      type=INTEGR, val=AUTH_HMAC_SHA_96
      type=PRF, val=PRF_HMAC_SHA
      type=DH_GROUP, val=1536.

Pre-existing IPsec VPN tunnels need to be cleared.

Should you need to clear an IKE gateway, use the following commands:

diagnose vpn ike restart
diagnose vpn ike gateway clear

LAN interface connection

To confirm whether a VPN connection over LAN interfaces has been configured correctly, issue a ping or traceroute command on the network behind the FortiGate unit to test the connection to a computer on the remote network. If the connection is properly configured, a VPN tunnel will be established automatically when the first data packet destined for the remote network is intercepted by the FortiGate unit.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel. You can confirm this by going to Monitor > IPsec Monitor where you will be able to see your connection. A green arrow means the tunnel is up and currently processing traffic. A red arrow means the tunnel is not processing traffic, and this VPN connection has a problem.

Dialup connection

A dialup VPN connection has additional steps. To confirm that a VPN between a local network and a dialup client has been configured correctly, at the dialup client, issue a ping command to test the connection to the local network. The VPN tunnel initializes when the dialup client attempts to connect.

If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This may or may not indicate problems with the VPN tunnel, or dialup client. As with the LAN connection, confirm the VPN tunnel is established by checking Monitor > IPsec Monitor.

Troubleshooting VPN connections

If you have determined that your VPN connection is not working properly through troubleshooting, the next step is to verify that you have a Phase2 connection.

If traffic is not passing through the FortiGate unit as you expect, ensure the traffic does not contain IPcomp packets (IP protocol 108, RFC 3173). FortiGate units do not allow IPcomp packets, they compress packet payload, preventing it from being scanned.

Testing Phase 1 and 2 connections is a bit more difficult than testing the working VPN. This is because they require diagnose CLI commands. These commands are typically used by Fortinet customer support to discover more information about your FortiGate unit and its current configuration.

Before you begin troubleshooting, you must:

• Configure FortiGate units on both ends for interface VPN
• Record the information in your VPN Phase 1 and Phase 2 configurations – for our example here the remote IP address is 10.11.101.10 and the names of the phases are Phase 1 and Phase 2
• Install a telnet or SSH client such as putty that allows logging of output
• Ensure that the admin interface supports your chosen connection protocol so you can connect to your FortiGate unit admin interface.

For this example, default values were used unless stated otherwise.

To get diagnose information for the VPN connection – CLI

1. Log into the CLI as admin with the output being logged to a file.
2. Stop any diagnose debug sessions that are currently running with the CLI command

diagnose debug disable

3. Clear any existing log-filters by running

diagnose vpn ike log-filter clear

4. Set the log-filter to the IP address of the remote computer (10.11.101.10). This filters out all VPN connections except ones to the IP address we are concerned with. The command is

diagnose vpn ike log-filter dst-addr4 10.11.101.10.

5. Set up the commands to output the VPN handshaking. The commands are:

diagnose debug app ike 255
diagnose debug enable

6. Have the remote FortiGate initiate the VPN connection in the web-based manager by going to VPN > IPsec Tunnels and selecting Bring up.
   This makes the remote FortiGate the initiator and the local FortiGate becomes the responder. Establishing the connection in this manner means the local FortiGate will have its configuration information as well as the information the remote computer sends. Having both sets of information locally makes it easier to troubleshoot your VPN connection.

7. Watch the screen for output, and after roughly 15 seconds enter the following CLI command to stop the output.

diagnose debug disable

8. If needed, save the log file of this output to a file on your local computer. Saving the output to a file can make it easier to search for a particular phrase, and is useful for comparisons.

To troubleshoot a phase1 VPN connection

Using the output from To get diagnose information for the VPN connection – CLI, search for the word proposal in the output. It may occur once indicating a successful connection, or it will occur two or more times for an unsuccessful connection — there will be one proposal listed for each end of the tunnel and each possible combination in their settings. For example if 10.11.101.10 selected both Diffie-Hellman Groups 1 and 5, that would be at least 2 proposals set.

A successful negotiation proposal will look similar to:

IPsec SA connect 26 10.12.101.10->10.11.101.10:500
config found
created connection: 0x2f55860 26 10.12.101.10->10.11.101.10:500
IPsec SA connect 26 10.12.101.10->10.11.101.10:500 negotiating
no suitable ISAKMP SA, queuing quick-mode request and initiating ISAKMP SA negotiation
initiator: main mode is sending 1st message...
cookie 3db6afe559e3df0f/0000000000000000
out [encryption]
sent IKE msg (ident-i1send): 10.12.101.10:500->10.11.101.10:500, len=264, id=3db6afe559e3df0f/0000000000000000
diaike 0: comes 10.12.101.1:500->10.11.101.1:500,ifindex=26....

Note the phrase “initiator: main mode is sending 1st message...” which shows you the handshake between the ends of the tunnel is in progress. Initiator shows the remote unit is sending the first message.

VPN troubleshooting tips

Attempting hardware offloading beyond SHA1

If you are trying to off-load VPN processing to a network processing unit (NPU), remember that only SHA1 authentication is supported. For high levels of authentication such as SHA256, SHA384, and SHA512 hardware offloading is not an option — all VPN processing must be done in software.

Enable/disable IPsec ASIC-offloading

Much like NPU-offload in IKE phase1 configuration, you can enable or disable the usage of ASIC hardware for IPsec Diffie-Hellman key exchange and IPsec ESP traffic. By default hardware offloading is used. For debugging purposes, sometimes it is best for all the traffic to be processed by software.

config sys global
   set ipsec-asic-offload [enable|disable]
end

Check Phase 1 proposal settings

Ensure that both sides have at least one Phase 1 proposal in common. Otherwise they will not connect. If there are many proposals in the list, this will slow down the negotiating of Phase 1. If its too slow, the connection may timeout before completing. If this happens, try removing some of the unused proposals.

NPU offloading is supported when the local gateway is a loopback interface.

Check your routing

If routing is not properly configured with an entry for the remote end of the VPN tunnel, traffic will not flow properly. You may need static routes on both ends of the tunnel. If routing is the problem, the proposal will likely setup properly but no traffic will flow.

Try enabling XAuth

If one end of an attempted VPN tunnel is using XAuth and the other end is not, the connection attempt will fail. The log messages for the attempted connection will not mention XAuth is the reason, but when connections are failing it is a good idea to ensure both ends have the same XAuth settings. If you do not know the other end’s settings enable or disable XAuth on your end to see if that is the problem.

General troubleshooting tips

Most connection failures are due to a configuration mismatch between the FortiGate unit and the remote peer. In general, begin troubleshooting an IPsec VPN connection failure as follows:

1. Ping the remote network or client to verify whether the connection is up.
2. Traceroute the remote network or client. If DNS is working, you can use domain names. Otherwise use IP addresses.
3. Check the routing behind the dialup client. Routing problems may be affecting DHCP. If this appears to be the case, configure a DHCP relay service to enable DHCP requests to be relayed to a DHCP server on or behind the FortiGate server.
4. Verify the configuration of the FortiGate unit and the remote peer. Check the following IPsec parameters:

• The mode setting for ID protection (main or aggressive) on both VPN peers must be identical.
• The authentication method (preshared keys or certificates) used by the client must be supported on the FortiGate unit and configured properly.
• If preshared keys are being used for authentication purposes, both VPN peers must have identical preshared keys.
• The remote client must have at least one set of Phase 1 encryption, authentication, and Diffie-Hellman settings that match corresponding settings on the FortiGate unit.
• Both VPN peers must have the same NAT traversal setting (enabled or disabled).
• The remote client must have at least one set of Phase 2 encryption and authentication algorithm settings that match the corresponding settings on the FortiGate unit.
• If you are using manual keys to establish a tunnel, the Remote SPI setting on the FortiGate unit must be identical to the Local SPI setting on the remote peer, and vise versa.

5. To correct the problem, see the following table.

VPN troubleshooting tips

A word about NAT devices

When a device with NAT capabilities is located between two VPN peers or a VPN peer and a dialup client, that device must be NAT traversal (NAT-T) compatible for encrypted traffic to pass through the NAT device.