1. Creating three virtual IP addresses

In this example, you open TCP ports 8096 (HTTP), 21 (FTP), and 22 (SSH) for remote users to communicate with the server behind the firewall. The external IP address of the server is 172.25.176.60, which is mapped to the internal IP address 192.168.70.10.

To create a virtual IP (VIP) address for port 8096, go to Policy & Objects > Virtual IPs and create a new virtual IP address.

Set External IP Address/Range to 172.25.176.60 and set Mapped IP Address/Range to 192.168.70.10.

Enable Port Forwarding. Set Protocol to TCP, set External Service Port to 80, and set Map to Port to 80.

Create a second VIP address for port 21. Set both External Service Port and Map to Port to 21.

Create a third VIP address for port 22. Set both External Service Port and Map to Port to 22.

2. Adding the virtual IP addresses to a VIP group

To add the new virtual IP addresses to a virtual IP group, go to Policy & Objects > Virtual IPs and create a new group.

Set the new virtual IP addresses as Members of the group.

3. Creating a security policy

To allow Internet users to reach the server, go to Policy & Objects > IPv4 Policy and create a new policy.

Set Incoming Interface to your Internet-facing interface, Outgoing Interface to the interface connected to the server, and Destination Address to the VIP group (webserver group).

NAT is disabled for this policy so that the server sees the original source addresses of the packets it receives. This is the preferred setting for a number of reasons. For example, the server logs are more meaningful if they record the actual source addresses of your users.

4. Results

To ensure that TCP port 8096 is open, browse to http://172.25.176.60:8096.

Next, ensure that TCP port 21 is open by using an FTP client to connect to the FTP server from a remote connection on the other side of the firewall.

Finally, ensure that TCP port 22 is open by connecting to the SSH server from a remote connection on the other side of the firewall.

1. Configuring the primary FortiGate

Connect to the primary FortiGate, click on the System Information dashboard widget and select Configure settings in System > Settings.

Change the Host name to identify this FortiGate as the primary FortiGate.

You can also enter this CLI command:

config system global
  set hostname Primary
end

Enter this CLI command to set the HA mode to active-passive, set a group id, group name and password, increase the device priority to a higher value (for example, 250) and enable override.

config system ha
  set mode a-p
  set group-id 100
  set group-name My-cluster
  set password <password>
  set priority 250
  set override enable
  set hbdev lan4 200 lan5 100
end

Enabling override and increasing the device priority means this FortiGate always becomes the primary unit.

This configuration also selects lan4 and lan5 to be the heartbeat interfaces and sets their priorities to 200 and 100 respectively. Its a best practice to set different priorities for the heartbeat interfaces (but not a requirement).

If you have more than one cluster on the same network, each cluster should have a different group id. Changing the group id changes the cluster interface virtual MAC addresses. If your group id causes a MAC address conflict on your network, you can select a different group id.

You can also configure most of these settings from the GUI (go to System > HA).

config system ha
  set group-id 100
  set override enable
end

After you enter the CLI command or make the GUI changes, the FortiGate negotiates to establish an HA cluster. You may temporarily lose connectivity with the FortiGate as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces are changed to HA virtual MAC addresses.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate unit (or just deleting all ARP table entries). You can usually delete the ARP table from a command prompt using a command similar to arp -d.

The FGCP uses virtual MAC addresses for failover. The virtual MAC address assigned to each FortiGate interface depends on the HA group ID. A group ID of 100 sets FortiGate interfaces to the following MAC addresses: 00:09:0f:09:64:00, 00:09:0f:09:64:01, 00:09:0f:09:64:02 and so on. For details, see Cluster virtual MAC addresses.

You can verify that the FGCP has set the virtual MAC addresses by viewing the configuration of each FortiGate interface from the GUI (go to Network > Interfaces) or by entering the following CLI command (shown below for lan2 on a FortiGate-51E):

get hardware nic lan2
... 
Current_HWaddr 00:09:0f:09:64:01 
Permanent_HWaddr 70:4c:a5:98:11:54
... 

You can also use the diagnose hardware deviceinfo nic lan2 command to display this information.

The output shows the current hardware (MAC) address (the virtual MAC set by the FGCP) and the permanent hardware (MAC) address for the interface.

2. Configuring the backup FortiGate

If required, change the firmware running on the new FortiGate to be the same version as is running on the primary FortiGate.

Enter the following command to reset the new backup FortiGate to factory default settings.

execute factoryreset

You can skip this step if the new FortiGate is fresh from the factory. But if its configuration has been changed at all, it’s a best practice to reset your FortiGate to factory defaults to reduce the chance of synchronization problems.

Register and apply licenses to the backup FortiGate before configuring it for HA operation. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, Security Rating, Outbreak Prevention, and additional virtual domains (VDOMs). All FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs. FortiToken licenses can be added at any time because they are synchronized to all cluster members.

Click on the System Information dashboard widget and select Configure settings in System > Settings. Change the FortiGate’s Host name to identify it as the backup FortiGate.

config system global
  set hostname Backup
end

Duplicate the primary unit HA settings, except set the Device Priority to a lower value (for example, 50) and do not enable override.

config system ha
  set mode a-p
  set group-id 100
  set group-name My-cluster
  set password <password>
  set priority 50
  set hbdev lan4 200 lan5 100
end

Similar to when configuring the primary FortiGate, once you enter the CLI command the backup FortiGate negotiates to establish an HA cluster. You may temporarily lose connectivity with the FortiGate as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces are changed to HA virtual MAC addresses.

If the group ID is the same, the backup FortiGate interfaces get the same virtual MAC addresses as the primary FortiGate. You can check Network > Interfaces on the GUI or use the get hardware nic command to verify.

3. Connecting the primary and backup FortiGates

Connect the primary and backup FortiGates together and to your network as shown in the network diagram at the start of the recipe. Making these connections disrupts network traffic as you disconnect and re-connect cables.

Switches must be used between the cluster and the Internet and between the cluster and the internal network as shown in the network diagram. You can use any good quality switches to make these connections. You can also use one switch for all of these connections as long as you configure the switch to separate traffic from the different networks.

The example shows the recommended configuration of direct connections between the lan4 heartbeat interfaces and between the lan5 heartbeat interfaces.

When the heartbeat interfaces are connected, the FortiGates find each other and negotiate to form a cluster. The primary FortiGate synchronizes its configuration to the backup FortiGate. The cluster forms automatically with minimal or no additional disruption to network traffic.

The cluster will have the same IP addresses as the primary FortiGate had. You can log into the cluster by logging into the primary FortiGate CLI or GUI using one of the original IP addresses of the primary FortiGate.

4. Checking cluster operation

Check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration. Log into the primary FortiGate CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If both cluster members have identical checksums you can be sure that their configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical you can use the information in Synchronizing the configuration to troubleshoot the problem or visit the Fortinet Support website for assistance.

The HA Status dashboard widget also shows synchronization status. Mouse over the host names of each FortiGate in the widget to verify that they are synchronized and both have the same checksum.

To view more information about the cluster status, click on the HA Status widget and select Configure Settings in System > HA (or go to System > HA).

5. Disabling override (recommended)

When the checksums are identical, disable override on the primary FortiGate by entering the following command:

config system ha
   set override disable
end

FGCP clusters dynamically respond to network conditions. If you keep override enabled, the same FortiGate always becomes the primary FortiGate. With override enabled; however, the cluster may negotiate more often to keep the same FortiGate as the primary FortiGate, potentially increasing traffic disruptions.

If you disable override it is more likely that the backup FortiGate could become the primary FortiGate. Disabling override is recommended unless its important that the same FortiGate remains the primary FortiGate.

6. Results

To test this, ping a reliable IP address from a PC on the internal network. After a moment, power off the primary FortiGate. You will see a momentary pause in the ping results, until traffic diverts to the backup FortiGate, allowing the ping traffic to continue.

64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=70 ttl=52 time=8.822 ms\
64 bytes from 184.25.76.114: icmp_seq=71 ttl=52 time=9.034 ms\
64 bytes from 184.25.76.114: icmp_seq=72 ttl=52 time=9.536 ms\
64 bytes from 184.25.76.114: icmp_seq=73 ttl=52 time=8.877 ms\
64 bytes from 184.25.76.114: icmp_seq=74 ttl=52 time=8.901 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=76 ttl=52 time=8.860 ms\
64 bytes from 184.25.76.114: icmp_seq=77 ttl=52 time=9.174 ms\
64 bytes from 184.25.76.114: icmp_seq=78 ttl=52 time=10.108 ms\
64 bytes from 184.25.76.114: icmp_seq=79 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=80 ttl=52 time=10.861 ms\
64 bytes from 184.25.76.114: icmp_seq=81 ttl=52 time=10.757 ms\
64 bytes from 184.25.76.114: icmp_seq=82 ttl=52 time=8.158 ms\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}

You can log into the cluster GUI or CLI using the same IP address as you had been using to the log into the primary FortiGate. If the primary FortiGate is powered off you will be logging into the backup FortiGate. Check the host name to verify the FortiGate that you have logged into. The FortiGate continues to operate in HA mode and if you restart the primary FortiGate, after a few minutes it should rejoin the cluster and operate as the backup FortiGate. Traffic should not be disrupted when the restarted primary unit rejoins the cluster.

1. Configure the Hub FortiGate

Using the CLI, configure phase 1 parameters.

The auto-discovery commands enable the sending and receiving of shortcut messages to spokes (the hub is responsible for lettings the spokes know that they should establish those tunnels).

Note: aggressive mode is not supported currently for ADVPN in 5.6, however support has been implemented starting in version 6.0.1.

config vpn ipsec phase1-interface
    edit "ADVPN"
        set type dynamic
        set interface "wan1"
        set proposal aes128-sha1
        set add-route disable
        set net-device enable
        set dhgrp 2
        set auto-discovery-sender enable
        set psksecret fortinet
    next
end

Configure the phase2 parameters.

This is a standard phase 2 configuration.

config vpn ipsec phase2-interface
    edit "ADVPN-P2"
        set phase1name "ADVPN"
        set proposal aes128-sha1
    next
end

Configure the tunnel interface IP.

ADVPN requires that tunnel IPs be configured on each device connecting to the topology. Those IP addresses need to be unique for each peer. A particularity of the hub is that it needs to define a bogus remote-IP address (10.10.10.254 in our example). This address should be unused in the topology and it will not be actually considered as part of the configuration for the hub.

config system interface
    edit "ADVPN"
        set vdom "root"
        set ip 10.10.10.1 255.255.255.255
        set type tunnel
        set remote-ip 10.10.10.254
        set interface "wan1"
    next
end

 Configure iBGP and route-reflection.

iBGP will be our overlay protocol of choice for enabling ADVPN communications. We are using an arbitrary private AS number (65000) in our example, and configuring a dynamic client group to reduce provisioning requirements.

While we are advertising our LAN network directly (“config network” command), route redistribution is a perfectly valid alternative.

config router bgp
    set as 65000
    set router-id 10.10.10.1
    config neighbor-group
        edit "ADVPN-PEERS"
            set remote-as 65000
            set route-reflector-client enable
            set next-hop-self enable
        next
    end
    config neighbor-range
        edit 0
            set prefix 10.10.10.0 255.255.255.0
            set neighbor-group "ADVPN-PEERS"
        next
    end
    config network
        edit 0
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

Configure basic policies to allow traffic to flow between the local network and the ADVPN VPN topology. As we generally desire traffic to be allowed between spokes in an ADVPN setup, we must remember to create a policy allowing spoke to spoke communications.

config firewall policy
    edit 0
        set name "OUT ADVPN"
        set srcintf "lan"
        set dstintf "ADVPN"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set status enable
    next
    edit 0
        set name "IN ADVPN"
        set srcintf "ADVPN"
        set dstintf "lan"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set status enable
    next
    edit 0
        set name "ADVPNtoADVPN"
        set srcintf "ADVPN"
        set dstintf "ADVPN"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set status enable
    next
end

2. Configure the Spoke FortiGates

Using the CLI, configure phase 1 parameters.

Note that we are configuring only one of the spokes in this example – the parameters that need to change for each spoke are highlighted in red.

config vpn ipsec phase1-interface
 edit "ADVPN"
  set interface "wan1"
  set proposal aes128-sha1
  set add-route disable
  set dhgrp 2
  set auto-discovery-receiver enable
  set remote-gw 10.1.1.1
  set psksecret fortinet
 next
end

Configure the phase2 parameters.

config vpn ipsec phase2-interface
    edit "ADVPN-P2"
        set phase1name "ADVPN"
        set proposal aes128-sha1
	set auto-negotiate enable
    next
end

 Configure the tunnel interface IP.

Notice that on the spokes, the remote IP is actually used and points to the IP defined on the hub.

config system interface
    edit "ADVPN"
        set vdom "root"
        set ip 10.10.10.2 255.255.255.255
        set type tunnel
        set remote-ip 10.10.10.1
        set interface "wan1"
    next
end

Config iBGP.

This is a static standard configuration and as stated for the hub, redistribution could be used instead of explicit route advertisement.

config router bgp
    set as 65000
    set router-id 10.10.10.2
    config neighbor
        edit "10.10.10.1"
            set soft-reconfiguration enable
            set remote-as 65000
            set next-hop-self enable
        next
    end
    config network
        edit 0
            set prefix 192.168.2.0 255.255.255.0
        next
    end
end

Configure a static route for the tunnel IP subnet.

This is an important special step for the spokes as they need a summary route that identifies all tunnel IP used over your topology to point towards the ADVPN interface. In our example, we use 10.10.10.0/24 (if our network planning expects less than 255 sites). Be sure to adequately plan this IP range as it needs to be hardcoded in the spokes.

config router static
    edit 0
        set dst 10.10.10.0 255.255.255.0
        set device "ADVPN"
    next
end

Configure policies.

config firewall policy
    edit 0
        set name "OUT ADVPN"
        set srcintf "lan"
        set dstintf "ADVPN"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set status enable
    next
    edit 0
        set name "IN ADVPN"
        set srcintf "ADVPN"
        set dstintf "lan"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set status enable
    next
end

Results

We can validate the behaviour of our configuration using a few commands. We are going to run these commands from SPOKE A.

get router info routing-table bgp will at a minimum display the learned routes from the topology. Note the recursive routing – a result of our spoke’s required static route. In this case, there has not been any traffic between our local subnet (192.168.2.0/24) and the other spoke’s subnet, as the routes are both going through the hub.

B       192.168.1.0/24 [200/0] via 10.0.0.1, ADVPN, 22:30:21
B       192.168.3.0/24 [200/0] via 10.0.0.3 (recursive via 10.0.0.1), 22:30:21

However once we initiate a ping between both spokes, we obtain a different display of routing information – routing now goes through a newly established dynamic tunnel directly through the remote spoke rather than through the hub. The ping hiccup that occurs is the tunnel rerouting through a newly negotiated tunnel to the other spoke.

Our routing information now displays the remote subnet as being available through the spoke directly, through interface ADVPN_0, a dynamically instantiated interface going to that spoke.

FG # exec ping-options source 192.168.2.1

FG # exec ping 192.168.3.1
PING 192.168.3.1 (192.168.3.1): 56 data bytes
64 bytes from 192.168.3.1: icmp_seq=0 ttl=254 time=38.3 ms
64 bytes from 192.168.3.1: icmp_seq=1 ttl=254 time=32.6 ms
Warning: Got ICMP 3 (Destination Unreachable)
64 bytes from 192.168.3.1: icmp_seq=2 ttl=255 time=43.0 ms
64 bytes from 192.168.3.1: icmp_seq=3 ttl=255 time=31.7 ms
64 bytes from 192.168.3.1: icmp_seq=4 ttl=255 time=31.2 ms

--- 192.168.3.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 31.2/35.3/43.0 ms

FG # get router info routing-table bgp 

B       192.168.1.0/24 [200/0] via 10.0.0.1, ADVPN, 22:34:13
B       192.168.3.0/24 [200/0] via 10.0.0.3, ADVPN_0, 00:02:28

FG # diag vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=ADVPN_0 ver=1 serial=a 10.1.1.2:0->10.1.1.3:0
bound_if=6 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/0
 parent=ADVPN index=0
proxyid_num=1 child_num=0 refcnt=19 ilast=3 olast=604 auto-discovery=2
stat: rxp=7 txp=7 rxb=1064 txb=588
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=ADVPN-P2 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA: ref=3 options=2f type=00 soft=0 mtu=1438 expire=42680/0B replaywin=2048 seqno=8 esn=0
  life: type=01 bytes=0/0 timeout=43152/43200
  dec: spi=9a487db3 esp=aes key=16 55e53d9fbc8dbeaa6df1032fbc80c4f6
       ah=sha1 key=20 a1470452c6a444f26a070add087f0d970c18e3a7
  enc: spi=3c37fea7 esp=aes key=16 8fd62a6745a9ba4fda062d4504b76851
       ah=sha1 key=20 44c606f1ef1bf5739ba62f6572031aa956974d0a
  dec:pkts/bytes=7/588, enc:pkts/bytes=7/1064
------------------------------------------------------
name=ADVPN ver=1 serial=9 10.1.1.2:0->10.1.1.1:0
bound_if=6 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/0
proxyid_num=1 child_num=1 refcnt=22 ilast=8 olast=8 auto-discovery=2
stat: rxp=3120 txp=3120 rxb=399536 txb=191970
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=12
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=ADVPN-P2 proto=0 sa=1 ref=2 serial=1 auto-negotiate adr
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA: ref=3 options=2f type=00 soft=0 mtu=1438 expire=4833/0B replaywin=2048 seqno=5ba esn=0
  life: type=01 bytes=0/0 timeout=43148/43200
  dec: spi=9a487db2 esp=aes key=16 4f70d27edad656cfcacbae61b23d4b11
       ah=sha1 key=20 b19ea87c90dd92d1cab58cbf24ae8fe12ee927cb
  enc: spi=b3dde355 esp=aes key=16 efbb4440df75018610b4ba8f5756167d
       ah=sha1 key=20 81cc9cee3bee1c2dba0eb1e7ac66e9d34b67bde9
  dec:pkts/bytes=1465/90152, enc:pkts/bytes=1465/187560
------------------------------------------------------

Enabling FortiMail Webmail Single Sign On

First we’ll need to enable single sign on for FortiMail webmail. Make sure your on Advanced Mode.

1. Go to System > Customization > Appearance.
2. Expand the Webmail Portal section.
3. Select “3rd Party/Single Sign on from the Login page drop down list. 
4. Select Edit.
5. Enter the Identity Provider (IDP) Metadata URL. A PortalGuard metadata URL is typically in the following format: https://yourservername.domain.com/sso/metadata.ashx
6. Copy the FortiMail server provider metadata URL on this same form
7. Open another browser and go to the URL to download the metadata file.

 Configuring PortalGuard Relying Party

Now to use the PortGuard server and configure PortalGuard relying party. 

1. Create a new Relying Party in the Identity Provider Configuration Editor.
2. Enter a Name & Description.
3. Enter the Identifier used by your FortiMail. This is the metadata file you downloaded in the previous section. 
4. Enter the Assertion Consumer URL for your FortiMail, which is typically in the format of https://yourservername.domain.com/sso/SAML2/POST (where yourservername.domain.com is the hostname and domain of the FortiMail – this can be found in the metadata file by searching for AssertionConsumerService Binding=”urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST” Location=)

 Creating Identity Claims

Now we’ll need to enter the Identity Claims tab and create a couple of new claims.

1. Create a new claim in the Identity Claims tab and enter the email address in the name section.
2. Select “http://schemas.xmlsoap.org/ws/2005/05/identity.claims.emailaddress” from the Pre-define Types button.
3. Select String Field from the Value Type dropdown menu and enter the field that holds the user email address in the Field Name. Select Save.
4. Create a second claim in the Identity Claims tab and enter the following information:
   – Enter EmailAddresstoNameID in the name section
   – Do not check the Send as NameID option
   – Enter urn:oid:0.9.2342.19200300.100.1.3 in the Scheme Type section. It must be manually entered
   – Select String Field from the Value Type dropdown menu
   – Enter “mail” or whichever fields holds the user email address in the Field Name section and then select Save.
5. Enter the remaining config information in the ldp-initiated and Authorization tabs
6. Save the Relying Party configuration and Apply To Identity Provider.

1. Creating tag categories and tags

In this example, you use tags to identify the following things about devices in the Security Fabric:

• Physical location
• Department
• Network administrators

To create the tag category for physical location, connect to Edge and go to System > Tags.

Set Tag Category to Location. Because each device in the network can only have one location, disable Allow multiple tag selection.

Add Tags for the first floor, second floor, and third floor.

Under Tag Scope, set Device to Mandatory.

For the department tag, enable Allow multiple tag selection.

Add Tags for the following departments: Accounting, Marketing, Sales, and Admin.

Under Tag Scope, set Interface to Mandatory and set Device to Mandatory. Because the FortiGate configuration includes default addresses, set Address to Optional.

For the network administrators tag, enable Allow multiple tag selection.

Add Tags for Robert and Lisa.

Under Tag Scope, set Device to Mandatory.

Because the configuration of tag categories and tags isn’t synchronized across  the Security Fabric, you must connect to each FortiGate device separately and add the appropriate tags for the part of your network that uses that FortiGate.

Connect to Accounting and repeat the previous steps to create the tags that are shown.

2. Applying tags to devices, interfaces, and addresses

To apply tags to devices in your network, go to User & Device > Device Inventory.

Edit the Accounting FortiGate.

Under Tags, add the following tags:

• For Department, add the Accounting tag
• For Location, add the Third floor tag 
• For Network administrators, add the Robert and Lisa tags

To apply tags to interfaces in your network, go to Network > Interfaces. Edit the interface that connects Edge and Accounting (in the example, port10).

Under Tags, set Department to Accounting.

To apply tags to addresses in your network, go to Policy & Objects > Addresses. Edit the address for the Accounting subnet.

Under Tags, set Department to Accounting.

To apply tags to devices in on the accounting network, connect to Accounting and go to User & Device > Device Inventory.

Edit a computer on this network.

Under Tags, add the following tags:

• For Department, add the Accounting tag 
• For Location, add the Third floor tag 
• For Network administrators, add the Robert tag

4. Results

To sort devices and interfaces by tags, connect to Edge and go to Security Fabric > Logical Topology.

In the Search field, enter Robert. The devices that have the Robert tag are highlighted.

To view more information about a highlighted device, including tags, hover over that device in the topology. The Robert tag is highlighted.

1. Connecting the FortiManager and Edge

In this example, port 16 on Edge connects to port 4 on the FortiManager.

To configure the interface on the root FortiGate, connect to Edge, go to Network > Interfaces, and edit port 16.

Configure Administrative Access to allow FMG-Access and FortiTelemetry.

To configure the interface on the FortiManager, connect to the FortiManager, go to System Settings > Network, select All Interfaces, and edit port4.

Set IP Address/Netmask to an internal IP address (in the example, 192.168.65.30/255.255.255.0).

Select Routing Table and add a default route for port 4. Set Gateway to the IP address of port 16 on Edge.

2. Allowing the FortiManager to have Internet access

To create an address for the FortiManager, connect to Edge, go to Policy & Objects > Addresses, and create a new address.

To allow the FortiManager to access the Internet, go to Policy & Objects > IPv4 Policy, and create a new policy.

3. Configuring central management

To enable central management, connect to Edge, go to Security Fabric > Settings, and enable Central Management.

Set Type to FortiManager, Mode to Normal, and set IP/Domain Name to the IP address of port 4 on the FortiManager.

After you select Apply, a message appears stating that the FortiManager received the message and Edge is waiting for management confirmation.

Edge, as the root FortiGate, pushes FortiManager settings to the other FortiGate devices in the Security Fabric. To verify this, connect to Accounting and go to Security Fabric > Settings.

To confirm the management connection, connect to the FortiManager and go to Device Manager > Unregistered Devices. Select the FortiGate devices and select + Add.

Connect to Edge. A warning message appears stating that the FortiGate is now managed by a FortiManager.

Select Login Read-Only.

Go to Security Fabric > Settings. Under Central Management, the Status is now Registered on FortiManager.

4. Results

The FortiGate devices are on the Managed FortiGate list and appear as part of a Security Fabric group. The * beside Edge indicates that it’s the root FortiGate in the Security Fabric.

Right-click on any of the FortiGate devices and select Fabric Topology. The topology of the Security Fabric is displayed.

1. Configuring the tunnel interfaces

To configure Edge to listen for FortiTelemetry traffic over the VPN, connect to Edge, go to Network > Interfaces, and edit the tunnel interface.

Set IP to the local IP address for this interface (10.10.10.1) and Remote IP/Network mask to the IP address for the Branch tunnel interface (10.10.10.2/32).

Under Administrative Access, enable FortiTelemetry.

Connect to Branch, go to Network > Interfaces, and edit the tunnel interface.

Set IP to the local IP address for this interface (10.10.10.2) and Remote IP/Network mask to the IP address for the Edge tunnel interface (10.10.10.1/32).

2. Adding the tunnel interfaces to the VPN

To create an address for the Edge tunnel interface, connect to Edge, go to Policy & Objects > Addresses, and create a new address.

Set Category to Address and set Subnet/IP Range to the IP address for the Edge tunnel interface (10.10.10.1/32).

Create a second address for the Branch tunnel interface. For this address, enable Static Route Configuration.

To allow VPN traffic between the Edge tunnel interface and the Branch tunnel interface, go to VPN > IPsec Tunnels, and edit the VPN tunnel. Select Convert To Custom Tunnel.

Under Phase 2 Selectors, create a new Phase 2. Set Local Address to use a Named Address and select the address for the Edge tunnel interface. Set Remote Address to use a Named Address, and select the address for the Branch tunnel interface.

To route traffic to the Branch tunnel interface, go to Network > Static Routes, and create a new route.

Set Destination to Named Address, and select the address for the Branch tunnel interface. Set Device to the tunnel interface.

To allow traffic between the tunnel interfaces, go to Policy & Objects > IPv4 Policy and edit the policy allowing local VPN traffic.

Set Source to include the Edge tunnel interface and Destination to include the Branch tunnel interface.

On Branch, repeat this step to include the following:

• Addresses for both tunnel interfaces (enable Static Route Configuration for the Branch tunnel interface address)
• A Phase 2 that allows traffic between the Branch tunnel interface and the Edge tunnel interface
• A static route to the Edge tunnel interface
• Edited policies that allow traffic to flow between the tunnel interfaces

To allow the new phase 2 to take effect, go to Monitor > IPsec Monitor, and restart the VPN tunnel.

3. Authorizing Branch for the Security Fabric

You can authorize a FortiGate, FortiAP, or FortiSwitch to join the Security Fabric by using the device’s serial number, rather than sharing the password for the Security Fabric.

To authorize Branch, connect to Edge, and enter the following CLI command:

config system csf
  config trusted-list
    edit <serial_number>
  end
end

To add Branch to the Security Fabric, connect to Branch, and go to Security Fabric > Settings. 

Enable FortiGate Telemetry. Set the Group name. Leave Group password blank. Enable Connect to upstream FortiGate. Set FortiGate IP to the IP address of the Edge tunnel interface.

To verify that Branch is now part of the Security Fabric, connect to Edge, and go to Security Fabric > Settings. Branch appears in the Topology.

4. Allowing Branch to access the FortiAnalyzer

To create an address for the FortiAnalyzer, connect to Branch, go to Policy & Objects > Addresses, and create a new address. Enable Static Route Configuration.

To allow VPN traffic between the FortiAnalyzer and the Branch tunnel interface, go to VPN > IPsec Tunnels, and create a new Phase 2.

To route traffic to the FortiAnalyzer, go to Network > Static Routes, and create a new route.

On Edge, repeat this step to create an address for FortiAnalyzer and a new Phase 2 that allows traffic between the FortiAnalyzer and the Branch tunnel interface. Edge doesn’t require a new static route.

To allow traffic between Branch and the FortiAnalyzer, go to Policy & Objects > IPv4 Policy, and create a new policy.

Set Incoming Interface to the VPN interface, and set Outgoing Interface to the interface that connects to the FortiAnalyzer (in the example, port16). Set Source to the Branch tunnel interface, and set Destination to the FortiAnalyzer.

Enable NAT for this policy.

To authorize the Branch FortiGate on the FortiAnalyzer, connect to the FortiAnalyzer, and go to Device Manager > Unregistered.

Select Branch, then select +Add to register Branch.

5. Results

To view Branch as part of the Security Fabric topology, connect to Edge and go to Security Fabric > Logical Topology. Branch is shown as part of the Security Fabric, connecting over the IPsec VPN tunnel.

6. Desynchronizing Branch from the FortiAnalyzer, FortiSandbox, and FortiManager (optional)

If you don’t want Branch to automatically use the settings that Edge pushes for the FortiAnalyzer, FortiSandbox, and FortiManager, use the following CLI command to configure these settings locally:

config system csf
  set configuration-sync local
end

Go to Security Fabric > Settings. You can now configure the settings for FortiAnalyzer logging, Central Management, and Sandbox Inspection. You can also choose to use local logging rather than sending logs to a FortiAnalyzer.

This option is available for all FortiGate devices in the Security Fabric, except for the root FortiGate.

1. Required information

Before you create a Fabric Connector for Microsoft Azure, you need to collect the following information:

• Azure tenant ID

• Azure client ID

• Azure client secret

• Azure subscription ID

• Azure resource group

All of this information is found through Azure rather than Fortinet sources. This recipe tells you where to find the information and where to apply it. Most of this recipe relates to a third party’s website, so some of this information, especially the Azure screenshots, could change. 

The following three items are already present in your Azure configuration and can be looked up easily.

2. Getting the Azure tenant ID

The Directory ID in Azure is the same as the Azure tenant ID required by the FortiGate.

Go to the Properties of the FortiGate in the Azure interface and find the Directory ID.

To do it through the CLI:

Step #A – Log into the Azure environment’s command line

$ azure login -v -u <user_id> -p <password>  
info:    Executing command login verbose: Authenticating... 
info:    Added subscription PAYG Dev/Test Subscription 
info:    Added subscription Fortinet Engineering 
info:    Added subscription Pay-As-You-Go 
info:    Added subscription SE-Subscription 
info:    login command OK $

 Step #B – Show the account information. Find the Tenant ID.

$ azure account show 
info:    Executing command account show 
data:    Name                        : Fortinet Engineering 
data:    ID                          : 2f96c44c-cfb2-4621-bd36-65ba45185e0c 
data:    State                       : Enabled 
data:    Tenant ID                   : 942b80cd-????-42a1-????-4b21dece61ba 
data:    Is Default                  : true 
data:    Environment                 : AzureCloud 
data:    Has Certificate             : No 
data:    Has Access Token            : Yes 
data:    User name                   : <user_id> 
data: 
info:    account show command OK

3. Getting the Azure subscription ID

The Azure subscription is referred to in the Azure interface as the Subscription ID.

Go to the Overview for the FortiGate VM resource.

In the Azure interface menu, under Favorites, select Dashboard, then from the All resources column, select the name of the FortiGate on which you are configuring the connector.

The Subscription ID is a hexadecimal number.

There are two variations of finding the Subscription ID information through the CLI:

1. Using an output table
2. Displaying a list.

First, use the same login procedure from of Obtaining the Azure Tenant ID through the CLI

Method A

i – Create an output table for the information

az account list --output table

ii – Display the table

$ az account show --subscription "<name of subscription>" | jq -r '.tenantId' 
942b80cd-????-42a1-????-4b21dece61ba <-- Directory ID $

or

Method B

$ az account list | jq -r '.[].tenantId' 

A few accounts are skipped as they don't have 'Enabled' state. Use '--all' to display them. 
942b80cd-????-42a1-????-4b21dece61ba 
942b80cd-????-42a1-????-4b21dece61ba 
942b80cd-????-42a1-????-4b21dece61ba 
$

4. Getting the Azure resource group

The Azure resource group is called the Resource group in the Azure interface.

The information is found on the same screen as the Subscription ID.

This value is a string value assigned by whoever configured the Resource group.

5. Azure client ID and Azure client secret

An Azure Active Directory application must be created to generate the Azure client ID and the corresponding Azure client secret, or Key as it is referred to in Azure. The complete instruction can be found at https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal.

Keep the following in mind when you get to the part about making a New application registration:

• The Application type has two options. Choose Web app/API. 
• The Sign-on URL has the asterisk commonly associated with a required field, but this is not applicable in this case. Put in any valid URL in the field to complete the form and enable the Create button.

The instructions show you how to find/create the two needed values, but different names are used.

• The field in the FortiGate interface called Azure client ID refers to the Application ID in Azure
• The field in the FortiGate interface called Azure secret refers to the Key for the application.

  6. Configuring the FortiGate

Once you have collected all of the values, enter them into the Fabric Connector form. It should look like this:

To do it through the CLI, create an sdn-connector and set the collected variables.

config system sdn-connector 
  edit "SF-connector1" 
    set status enable 
    set type azure 
    set tenant-id "942b8<partially removed for security>dece61ba" 
    set subscription-id "4f27<partially removed for security>e2f9a" 
    set client-id "f66d<partially removed for security>b0755" 
    set client-secret ENC cqAmO4/d<partially removed for security>cZJ4wyAZOQ== 
    set resource-group "<user id>grp001" 
    set azure-region global 
    set update-interval 60 
  end 

Verification of permissions on the application

At the end of Azure’s documentation for registering the app, there is a section on  Assign application to role. Be sure to do this as well.

  7. Tagging resources

In preparation for showing how to create a dynamic address, an instance of a Linux system was created and tagged with the Name “Owner” and the value “test”. To do this for your own resources:

• Select the resource
• Go to the Overview
• In the Overview section, edit the Tags.

You can create customized tag names and you can have more than 1 tag associated with a resource.

  8. Creating an address

In order to confirm that the connector has been successfully configured, you need to have a Fabric Connector Address.

• The address or address group is used for source/destination of firewall policies. The Address is based on IP addresses. The address contains address(es) within the Azure instance.
• When changes occur to addresses in the Azure environment, the Fabric Connector populates and updates the changes automatically based on the specified filtering condition so administrators do not need to reconfigure the Address’s content manually.
• As instances that match the filter appear in the environment, changes are propagated to the firewall policies that use the address object.

Configuring one of these addresses is similar to configuring any other address object, but with a few different options.

• Go to Policy & Objects > Addresses
• Give the address a name
• For the Type field, select Fabric Connector Address from the drop-down menu
• For the Fabric Connector Type select Microsoft Azure from the drop-down menu
• Input a filter into the Filter field
• Set the Interface to a specific port or leave it at the default “any”
• Add any Comments or Tags that are applicable

The CLI version of the same address would be:

config firewall address 
  edit "azure-client" 
    set type dynamic   
    set comment ''   
    set visibility enable   
    set associated-interface ''   
    set color 0   
    set sdn azure   
    set filter "tag.Owner=test" 
  next 
end

Filters

Tags are not the only option to filter the address. The Azure Fabric Connector supports the following filters:

• vm=<vm id>
• securitygroup=<nsg id>
• vnet=<virtual network id>
• subnet=<subnet id>
• vmss=<vmss id>
• tag.<key>=<value>

Just like the tag value, these properties are found in the Azure interface

9. Dynamic address in a Policy

That same policy looks like this in the CLI:

config firewall policy 
  edit 0 
    set name "outgoing1"   
    set srcintf "port2"   
    set dstintf "port1"   
    set srcaddr "azure-client"   
    set dstaddr "all"   
    set action accept   
    set schedule "always"   
    set service "ALL"   
    set logtraffic all   
    set logtraffic-start enable   
    set capture-packet enable   
    set nat enable 
  next 
end

1. Configuring the IPsec VPN on HQ

To create a new IPsec VPN tunnel, connect to HQ, go to VPN > IPsec Wizard, and create a new tunnel.

In the VPN Setup step, set Template Type to Site to Site, set Remote Device Type to FortiGate, and set NAT Configuration to No NAT between sites.

In the Authentication step, set IP Address to the public IP address of the Branch FortiGate (in the example, 172.25.177.46).

After you enter the IP address, the wizard automatically assigns an interface as the Outgoing Interface. If you want to use a different interface, select it from the drop-down menu.

Set a secure Pre-shared Key.

In the Policy & Routing step, set Local Interface to lan. The wizard adds the local subnet automatically. Set Remote Subnets to the Branch network’s subnet (in the example, 192.168.13.0/24).

Set Internet Access to None.

A summary page shows the configuration created by the wizard, including interfaces, firewall addresses, routes, and policies.

To view the VPN interface created by the wizard, go to Network > Interfaces.

To view the firewall addresses created by the wizard, go to Policy & Objects > Addresses.

To view the routes created by the wizard, go to Network > Static Routes.

To view the policies created by the wizard, go to Policy & Objects > IPv4 Policy.

2. Configuring the IPsec VPN on Branch

To create a new IPsec VPN tunnel, connect to Branch, go to VPN > IPsec Wizard, and create a new tunnel.

In the VPN Setup step, set Template Type to Site to Site, set Remote Device Type to FortiGate, and set NAT Configuration to No NAT between sites.

In the Authentication step, set IP Address to the public IP address of the HQ FortiGate (in the example, 172.25.176.62).

After you enter the IP address, the wizard automatically assigns an interface as the Outgoing Interface. If you want to use a different interface, select it from the drop-down menu.

Set the secure Pre-shared Key that was used for the VPN on HQ.

In the Policy & Routing step, set Local Interface to lan. The wizard adds the local subnet automatically. Set Remote Subnets to the HQ network’s subnet (in the example, 192.168.65.0/24).

Set Internet Access to None.

A summary page shows the configuration created by the wizard, including interfaces, firewall addresses, routes, and policies.

To bring the VPN tunnel up, go to Monitor > IPsec Monitor. Right-click under Status and select Bring Up.

3. Results

Users on the HQ internal network can access resources on the Branch internal network and vice versa.

To test the connection, ping HQ’s LAN interface from a device on the Branch internal network.

Creating an Impersonation Analysis Profile

First you will need to create an impersonation profile and add display names and email addresses to map.

1. Go to Profile > AntiSpam > Impersonation.
2. Select New to create a new profile.
3. Enter an appropriate profile name.
4. Select a domain from the Domain dropdown list. 
5. Select New in the Impersonation Entry section to add individuals within your business you know to be safe.
6. Enter the display name to be mapped to the email address.
7. Select Wildcard or Regular expression from the Pattern type dropdown list.
8. Enter the email address to be mapped to the display name.
9. Select Create and then Create once more.

Activating the Impersonation Profile

Now you’ll need to enable impersonation analysis in the antispam profile to check for mapping and then select the profile.

1. Go to Profile > AntiSpam > AntiSpam.
2. Select New or modify an existing profile.
3. Expand the Scan Configurations section and enable Impersonation analysis.
4. Select the desired action you wish the unit to take when it encounters the problem.
5. Select either Create or OK.
6. When you create an IP policy or recipient policy, choose the antispam profile that contains the impersonation analysis profile.

Configuring Dynamic Scanning

In addition to manually entering mapping entries and creating impersonation analysis profiles, FortiMail Mail Statistics Service can automatically learn and track the mapping.

To use the FortiMail manual/dynamic, or both,  impersonation analysis scanning, enter the following command:

config antispam settings
set impersonation-analysis dynamic manual

By default, FortiMail uses manual analysis only.

Also enable the FortiMail Mail Statistics Service with the following command. This service is also disabled by default:

config system global
set mailstat-service enable

1. Configuring clustering

If required, upgrade the firmware running on both FortiGates. Both FortiGates should be running the same version of FortiOS.

On each FortiGate, enter the following command to reset it to factory default settings.

execute factoryreset

You can skip this step if the FortiGates are fresh from the factory. But if their configurations have changed at all, it’s a best practice to reset them to factory defaults to reduce the chance of synchronization problems.

Change the primary FortiGate Host name to identify it as the primary FortiGate by going to System > Settings.

Change the backup FortiGate Host name to identify it as the backup FortiGate by going to System > Settings.

You can also use the CLI to change the host name. From the Primary FortiGate:

config system global
  set hostname Primary
end

From the Backup FortiGate:

config system global
  set hostname Backup
end

Register and apply licenses to both FortiGates before forming the cluster. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, Security Rating, Outbreak Prevention, and additional virtual domains (VDOMs).

Both FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs.

You can add FortiToken licenses and install third-party certificates at any time because they’re synchronized to all cluster members.

On the primary FortiGate, enter the following CLI command to set the HA mode to active-passive, set a group-id, group name, and password, increase the device priority to 200, enable override, and configure the heartbeat interfaces (lan4 and lan5 in this example).

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 200
  set override enable
  set hbdev lan4 200 lan5 100
end

You can also configure most of these settings from the GUI (go to Global > System > HA). The group-id and override can only be configured from the CLI.

On the backup FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 50.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 50
  set override enable
  set hbdev lan4 200 lan5 100
end

After you enabe HA, the FortiGates negotiate to establish an HA cluster. You may temporarily lose connectivity as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces change to HA virtual MAC addresses.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate (or just deleting all ARP table entries). You can usually delete the ARP table from a command prompt using a command similar to arp -d.

The FGCP uses virtual MAC addresses for failover. The virtual MAC address assigned to each FortiGate interface depends on the HA group ID. A group ID of 88 sets FortiGate interfaces to the following MAC addresses: 00:09:0f:09:58:00, 00:09:0f:09:58:01, 00:09:0f:09:58:02 and so on. For details, see Cluster virtual MAC addresses.

You can verify that the FGCP has set the virtual MAC addresses by viewing the configuration of each FortiGate interface from the GUI (go to Network > Interfaces) or by entering the following CLI command (shown below for lan2 on a FortiGate-51E):

get hardware nic lan2
... 
Current_HWaddr 00:09:0f:09:58:01 
Permanent_HWaddr 70:4c:a5:98:11:54
... 

You can also use the diagnose hardware deviceinfo nic lan2 command to display this information.

The output shows the current hardware (MAC) address (the virtual MAC set by the FGCP) and the permanent hardware (MAC) address for the interface.

2. Connecting and verifying cluster operation

Connect the primary and backup FortiGates together and to your networks as shown in the network diagram at the start of the recipe. Making these connections disrupts network traffic as you disconnect and re-connect cables.

Switches must be used between the cluster and the Internet, between the cluster and the internal network, and between the cluster and the Engineering network as shown in the diagram. You can use any good quality switches to make these connections. You can also use one switch for all of these connections as long as you configure the switch to separate traffic from the different networks.

You can connect the heartbeat interfaces (lan4 and lan5) with direct cable connections as shown.

When you connect the heartbeat interfaces and power on the FortiGates, they find each other and negotiate to form a cluster. The cluster will have the same IP addresses as the primary FortiGate. You can log into the cluster by logging into the primary FortiGate GUI or CLI using one of the original IP addresses of the primary FortiGate.

Check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration. Log into the primary FortiGate CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If both cluster members have identical checksums you can be sure that their configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical you can use the information in Synchronizing the configuration to troubleshoot the problem or visit the Fortinet Support website for assistance.

You can also use the get system ha status command to display detailed information about the cluster. For information about this command, see Viewing cluster status from the CLI for details.

The HA Status dashboard widget widget also shows synchronization status. Hover over the host names of each FortiGate in the widget to verify that they are synchronized and both have the same checksum.

3. Adding VDOMs and setting up virtual clustering

Enable VDOMs by going to System > Settings > System Operation Settings and enabling Virtual Domains. Or enter the following CLI command.

config system global
  set vdom-admin enable
end

Add VDOMs as required. Go to Global > System > VDOM and select Create New. Or enter the following CLI command to add the Engineering VDOM.

config global
  edit Engineering
end

Configure virtual clustering and VDOM partitioning on the primary FortiGate. The following command enables virtual cluster 2, adds the Engineering VDOM to virtual cluster 2, and sets the virtual cluster 2 device priority of the primary FortiGate to 50.

config global
  config system ha
    set vcluster2 enable
      config secondary-vcluster
        set vdom Engineering
        set priority 50
      end

You can also configure virtual clustering and VDOM partitioning from the GUI by going to Global > System > HA.

Set the virtual cluster 2 priority of the backup FortiGate to a relatively high value (in this example, 200) so that the backup FortiGate processes traffic for the VDOMs in virtual cluster 2. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the backup FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 200
  end

4. Checking virtual cluster operation

Once again use the diagnose sys ha checksum cluster command and the get system ha status command to check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration.

The HA Status dashboard widget shows the VDOMs in the virtual clusters. You can hover over the VDOM names to see status information for the VDOMs. You can hover over the host names of each FortiGate to verify that they are synchronized and both have the same checksum.

To view more information about the cluster status, click on the HA Status widget and select Configure Settings in System > HA (or go to System > HA).

The HA status page shows both FortiGates in the cluster. It also shows that Primary is the primary (master) FortiGate for the root VDOM (so the primary FortiGate processes all root VDOM traffic). The page also shows that Backup is the primary (Master) FortiGate for the Engineering VDOM (so the backup FortiGate processes all Engineering VDOM traffic).

7. Results

All root VDOM traffic should now be flowing through the primary FortiGate and Engineering VDOM traffic should be flowing through the backup FortiGate. If the primary FortiGate becomes unavailable, the cluster negotiates and traffic fails over and all traffic would be processed by the backup FortiGate.

To test failover, ping a reliable IP address from a PC on the internal or Engineering network. After a moment, power off the primary FortiGate. You will see a momentary pause in the ping results while the cluster negotiates and then ping traffic can continue.

64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=70 ttl=52 time=8.822 ms\
64 bytes from 184.25.76.114: icmp_seq=71 ttl=52 time=9.034 ms\
64 bytes from 184.25.76.114: icmp_seq=72 ttl=52 time=9.536 ms\
64 bytes from 184.25.76.114: icmp_seq=73 ttl=52 time=8.877 ms\
64 bytes from 184.25.76.114: icmp_seq=74 ttl=52 time=8.901 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=76 ttl=52 time=8.860 ms\
64 bytes from 184.25.76.114: icmp_seq=77 ttl=52 time=9.174 ms\
64 bytes from 184.25.76.114: icmp_seq=78 ttl=52 time=10.108 ms\
64 bytes from 184.25.76.114: icmp_seq=79 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=80 ttl=52 time=10.861 ms\
64 bytes from 184.25.76.114: icmp_seq=81 ttl=52 time=10.757 ms\
64 bytes from 184.25.76.114: icmp_seq=82 ttl=52 time=8.158 ms\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}

You can log into the cluster GUI or CLI using the same IP address as you had been using to the log into the primary FortiGate. If the primary FortiGate is powered off you will be logging into the backup FortiGate. Check the host name to verify the FortiGate that you have logged into.

When you restart the primary FortiGate, after a few minutes it should rejoin the cluster and because override is enabled, the original virtual cluster configuration should be re-established. Traffic may be temporarily disrupted when the restarted primary FortiGate rejoins the cluster.

1. Preparing the FortiGates

If required, upgrade the firmware running on the FortiGates. All of the FortiGates should be running the same version of FortiOS.

On each FortiGate, enter the following command to reset them factory default settings.

execute factoryreset

You can skip this step if the FortiGates are fresh from the factory. But if their configurations have changed at all, it’s a best practice to reset them to factory defaults to reduce the chance of synchronization problems.

Change the primary FortiGate Host name to identify it as the primary FortiGate by going to System > Settings.

Change the backup FortiGate Host name to identify it as Backup-1 by going to System > Settings.

Change the third FortiGate Host name to identify it as Backup-2 by going to System > Settings.

Change the fourth FortiGate Host name to identify it as Backup-3 by going to System > Settings.

You can also use the CLI to change the host name. From the Primary FortiGate:

config system global
  set hostname Primary
end

From the Backup-1 FortiGate:

config system global
  set hostname Backup-1
end

From the Backup-2 FortiGate:

config system global
  set hostname Backup-2
end

From the Backup-3 FortiGate:

config system global
  set hostname Backup-3
end

Register and apply licenses to the FortiGates before forming the cluster. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, Security Rating, Outbreak Prevention, and additional virtual domains (VDOMs).

All of the FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs.

You can add FortiToken licenses and install third-party certificates at any time because they’re synchronized to all cluster members.

2. Configuring clustering

On the primary FortiGate, enter the following CLI command to set the HA mode to active-passive, set a group-id, group name, and password, increase the device priority to 200, enable override, and configure the heartbeat interfaces (lan4 and lan5 in this example).

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 200
  set override enable
  set hbdev lan4 200 lan5 100
end

You can also configure most of these settings from the GUI (go to Global > System > HA). The group-id and override can only be configured from the CLI.

On the Backup-1 FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 50. Setting the device priority to a relatively low value means the Backup-1 FortiGate will most likely always become the backup FortiGate.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 50
  set override enable
  set hbdev lan4 200 lan5 100
end

On the Backup-2 FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 150. A device priority of 150 is almost as high as the device priority of the primary FortiGate. So if the primary FortiGate fails, the Backup-2 FortiGate should become the new primary FortiGate.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 150
  set override enable
  set hbdev lan4 200 lan5 100
end

On the Backup-3 FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 100. A device priority of 100 means that if the backup FortiGate fails, the Backup-3 FortiGate will have the lowest device priority so will become the new backup FortiGate.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 100
  set override enable
  set hbdev lan4 200 lan5 100
end

After you enable HA, each FortiGate negotiates to establish an HA cluster. You may temporarily lose connectivity as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces change to HA virtual MAC addresses.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate (or just deleting all ARP table entries). You can usually delete the ARP table from a command prompt using a command similar to arp -d.

The FGCP uses virtual MAC addresses for failover. The virtual MAC address assigned to each FortiGate interface depends on the HA group ID. A group ID of 88 sets FortiGate interfaces to the following MAC addresses: 00:09:0f:09:58:00, 00:09:0f:09:58:01, 00:09:0f:09:58:02 and so on. For details, see Cluster virtual MAC addresses.

You can verify that the FGCP has set the virtual MAC addresses by viewing the configuration of each FortiGate interface from the GUI (go to Network > Interfaces) or by entering the following CLI command (shown below for lan2 on a FortiGate-51E):

get hardware nic lan2
... 
Current_HWaddr 00:09:0f:09:58:01 
Permanent_HWaddr 70:4c:a5:98:11:54
... 

You can also use the diagnose hardware deviceinfo nic lan2 command to display this information.

The output shows the current hardware (MAC) address (the virtual MAC set by the FGCP) and the permanent hardware (MAC) address for the interface.

3. Connecting and verifying cluster operation

Connect the FortiGates together and to your networks as shown in the network diagram at the start of the recipe. Making these connections disrupts network traffic as you disconnect and re-connect cables.

Switches must be used between the cluster and the Internet, between the cluster and the internal network, and between the cluster and the Engineering network as shown in the diagram. You can use any good quality switches to make these connections.

To make HA heartbeat connections, connect all of the lan4 interfaces to the same switch and all of the lan5 interfaces to another switch.

You can also use fewer switches for all of these connections as long as you configure the switches to separate traffic from the different networks.

When you connect the heartbeat interfaces and power on the FortiGates, they find each other and negotiate to form a cluster. The cluster will have the same IP addresses as the primary FortiGate. You can log into the cluster by logging into the primary FortiGate GUI or CLI using one of the original IP addresses of the primary FortiGate.

Check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration. Log into the primary FortiGate CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If both cluster members have identical checksums you can be sure that their configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical you can use the information in Synchronizing the configuration to troubleshoot the problem or visit the Fortinet Support website for assistance.

You can also use the get system ha status command to display detailed information about the cluster. For information about this command, see Viewing cluster status from the CLI for details.

The HA Status dashboard widget widget also shows synchronization status. Hover over the host names of each FortiGate in the widget to verify that they are synchronized and both have the same checksum.

4. Adding VDOMs and setting up virtual clustering

Enable VDOMs by going to System > Settings > System Operation Settings and enabling Virtual Domains. Or enter the following CLI command.

config system global
  set vdom-admin enable
end

Add VDOMs as required. Go to Global > System > VDOM and select Create New. Or enter the following CLI command to add the Engineering VDOM.

config global
  edit Engineering
end

Configure virtual clustering and VDOM partitioning on the primary FortiGate. The following command enables virtual cluster 2, adds the Engineering VDOM to virtual cluster 2, and sets the virtual cluster 2 device priority of the primary FortiGate to 50.

config global
  config system ha
    set vcluster2 enable
      config secondary-vcluster
        set vdom Engineering
        set priority 50
      end

You can also configure virtual clustering and VDOM partitioning from the GUI by going to Global > System > HA.

Set the virtual cluster 2 priority of the Backup-1 FortiGate to a relatively high value (in this example, 200) so that this FortiGate processes traffic for the VDOMs in virtual cluster 2. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the backup FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 200
  end

Set the virtual cluster 2 priority of the Backup-2 FortiGate to 100 so that if the primary FortiGate fails, Backup-2 will become the primary FortiGate but will have the lowest virtual cluster 2 priority. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the Backup-2 FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 100
  end

Set the virtual cluster 2 priority of the Backup-3 FortiGate to 150 so that if the backup FortiGate fails, Backup-3 will have the highest virtual cluster 2 device priority. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the backup FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 150
  end

5. Checking virtual cluster operation

Once again use the diagnose sys ha checksum cluster command and the get system ha status command to check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration.

The HA Status dashboard widget shows the VDOMs in the virtual clusters. You can hover over the VDOM names to see status information for the VDOMs. You can hover over the host names of each FortiGate to verify that they are synchronized and both have the same checksum.

To view more information about the cluster status, click on the HA Status widget and select Configure Settings in System > HA (or go to System > HA).

The HA status page shows all four FortiGates in the cluster. It also shows that Primary is the primary (master) FortiGate for the root VDOM (so the primary FortiGate processes all root VDOM traffic). The page also shows that Backup-1 is the primary (master) FortiGate for the Engineering VDOM (so the backup FortiGate processes all Engineering VDOM traffic).

6. Results

All root VDOM traffic should now be flowing through the primary FortiGate and Engineering VDOM traffic should be flowing through the backup FortiGate. If the primary FortiGate becomes unavailable, the cluster negotiates and traffic fails over and all traffic would be processed by the backup FortiGate.

To test failover, ping a reliable IP address from a PC on the internal or Engineering network. After a moment, power off the primary FortiGate. You will see a momentary pause in the ping results while the cluster negotiates and then ping traffic can continue.

64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=70 ttl=52 time=8.822 ms\
64 bytes from 184.25.76.114: icmp_seq=71 ttl=52 time=9.034 ms\
64 bytes from 184.25.76.114: icmp_seq=72 ttl=52 time=9.536 ms\
64 bytes from 184.25.76.114: icmp_seq=73 ttl=52 time=8.877 ms\
64 bytes from 184.25.76.114: icmp_seq=74 ttl=52 time=8.901 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=76 ttl=52 time=8.860 ms\
64 bytes from 184.25.76.114: icmp_seq=77 ttl=52 time=9.174 ms\
64 bytes from 184.25.76.114: icmp_seq=78 ttl=52 time=10.108 ms\
64 bytes from 184.25.76.114: icmp_seq=79 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=80 ttl=52 time=10.861 ms\
64 bytes from 184.25.76.114: icmp_seq=81 ttl=52 time=10.757 ms\
64 bytes from 184.25.76.114: icmp_seq=82 ttl=52 time=8.158 ms\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}

You can log into the cluster GUI or CLI using the same IP address as you had been using to the log into the primary FortiGate. If the primary FortiGate is powered off you will be logging into the Backup-1 FortiGate. Check the host name to verify the FortiGate that you have logged into.

After the primary FortiGate fails the HA Status dashboard widget shows that the Backup-2 has become the primary (master) FortiGate.

The System > HA page shows that the Backup-2 FortiGate has become the primary FortiGate for virtual cluster 1. This page also shows that the Backup-1 FortiGate continues to process virtual cluster 2 traffic.

If you restart the primary FortiGate, after a few minutes it should rejoin the cluster and because override is enabled, the original virtual cluster configuration should be re-established. Traffic may be temporarily disrupted when the restarted primary FortiGate rejoins the cluster.

You can also try powering off other FortiGates in the virtual cluster to see how the cluster adapts to the failover. Because of the device priority configuration, if two FortiGates are operating, virtual cluster 1 and virtual cluster 2 traffic will be distributed between them.